Build the Horse, Not the Car

A drone safety debate with lessons for AI agent design

By Chad Butler | Published April 2026

What Happens When You Remove the Pilot and Keep the Same Controls?

Answer: The controls fail. When the Commercial Drone Alliance asked the FAA to skip onboard detect-and-avoid for autonomous drones, they proposed a safety model that broadcasts the location and heading data attackers need to hijack a drone's GPS, with no independent system to catch the attack.

Why this matters: This is not just a drone problem. It is the same design mistake teams make when building AI agents and autonomous systems. They take controls that worked when a human was in the loop and assume those controls hold up when the human is gone. If you are a CISO or security leader reviewing designs for agentic applications, this story is a case study in what goes wrong when you skip that question.

▶️ Prefer video? I break this down in my YouTube video: I Led Security for Amazon's Drones. The Industry Has a Problem.

What you'll get:

  • A clear understanding of why Amazon pulled out of the Commercial Drone Alliance

  • A practical mental model for auditing controls in autonomous or agentic system

  • The one question you should be asking in every design review for systems that remove humans from the loop

Common pitfalls:

  • Assuming a control that works with human oversight will work without it

  • Treating broadcast or logging as a substitute for independent verification

  • Skipping threat modeling when transitioning from human-in-the-loop to autonomous operation

Why did Amazon leave the Commercial Drone Alliance?

Answer: Amazon left the Commercial Drone Alliance because the Alliance is pushing a broadcast-only safety model that removes the requirement for drones to independently detect and avoid obstacles.

Context and principles: The FAA is writing the rules for BVLOS (Beyond Visual Line of Sight) drone operations. This is when a drone flies autonomously, out of the operator's sight, making its own navigation decisions. Two approaches are competing for how to conduct these flights safely.

The Commercial Drone Alliance wants all aircraft below 500 feet to broadcast their position, speed, and heading using ADS-B or similar protocols. Their logic is simple: if everyone announces where they are, everyone can avoid each other. They argue that onboard detect-and-avoid sensors (radar, cameras, LIDAR) are too heavy, too expensive, and impractical for small delivery drones.

Amazon's position is different. Amazon says drones need onboard detect-and-avoid. Cameras, radar, and sensors on the drone itself that can independently see obstacles, regardless of what other aircraft are broadcasting. Amazon left the Alliance because they believe the broadcast-only approach is not safe enough. And they have data to back it up.

Over 70,000 flights, Amazon's Prime Air detect-and-avoid system performed successful collision avoidance on two potential mid-air collisions with other aircraft. One of those was a helicopter. That helicopter was legally required to broadcast its position over ADS-B. It wasn't broadcasting. Under a broadcast-only safety model, the drone would have had no idea the helicopter was there. The onboard sensors caught what the broadcast system missed.

That is the difference between a system that depends on everyone else following the rules and a system that can independently verify what is actually in front of it.

Key takeaway: A broadcast-only safety model fails when a single aircraft does not comply. Independent detect-and-avoid does not have that dependency.

How does broadcast-only navigation create a GPS spoofing attack surface?

Answer: Requiring drones to continuously broadcast precise position, speed, and heading on unencrypted channels hands an attacker the data they need to covertly hijack the drone's GPS.

The hardest part of GPS spoofing is not generating a fake signal. The hard part is making the fake signal match what the drone's navigation system already expects, so the drone doesn't notice the switch. To do that covertly, the attacker needs real-time position and velocity data.

Research from the UT Austin Radionavigation Lab showed that covert GPS capture requires precise knowledge of the target's position and velocity, and that the spoofer could obtain this data by intercepting ADS-B messages. Both ADS-B and Remote ID transmit on unencrypted, unauthenticated channels. No encryption, no authentication, no integrity verification. And this situation is not likely to change soon.

Without that broadcast data, a spoofing attempt is noisy. The drone's Extended Kalman Filter detects the mismatch and triggers a failsafe. With the broadcast data, the attacker generates a signal that matches within the required tolerances (80 nanoseconds of time offset and 500 meters of location offset), and the drone doesn't flag an anomaly.

Attack sequence:

  1. Attacker passively collects broadcast position, speed, and heading using a cheap SDR receiver or smartphone app. No active scanning required.

  2. Attacker configures a GPS spoofer to generate counterfeit signals precisely aligned with what the drone's receiver expects

  3. Attacker gradually overpowers the legitimate GPS signal while maintaining alignment. The navigation system never flags the switch.

  4. Attacker slowly introduces positional offsets, steering the drone off course

  5. Without onboard detect-and-avoid, no independent sensor exists to cross-check the GPS data

Context: GPS signals travel over 12,000 miles from satellites in orbit. By the time they reach the drone, they are extremely weak. A low-power SDR transmitting from the ground at close range can overpower them with very little effort.

Without broadcast data, this attack is hard. The attacker has to independently figure out where the drone is and how fast it is moving. And if you get it wrong, the drone's navigation system detects the mismatch and triggers a failsafe. The attack is noisy and easy to catch.

With mandatory broadcast, the attack becomes much easier. The attacker sits back, passively collects the drone's telemetry, aligns the spoofed signal to match expected values, and takes control without triggering any alert. The drone flies wherever the attacker wants.

This is not theoretical. The UT Austin team demonstrated successful GPS capture of UAVs using broadcast telemetry data. The CDA's proposal would mandate the broadcast of this data by regulation.

Key takeaway: The CDA is proposing a system that mandates the broadcast of the telemetry data an attacker needs to hijack a drone, while opposing the primary defense that would catch the attack and prevent collisions.

What can AI Agentic App Builders Learn From This?

Ask one question in every design review: does this control still work without a human in the loop?

In traditional aviation, pilots also broadcast their position over ADS-B. But GPS spoofing does not cause a human pilot to fly into a school. The pilot can cross-check instruments, look out the window, or radio air traffic control. The pilot is the final detect-and-avoid system.

When you remove the pilot and go autonomous, every control that silently depended on human judgment becomes a single point of failure. The broadcast system was one layer in a stack that included a trained human with eyes, instruments, and judgment. It was never designed to be the only safety layer.

And broadcast compliance is not guaranteed. In January 2025, a U.S. Army Black Hawk helicopter collided with an American Airlines regional jet over the Potomac River. The helicopter was not broadcasting ADS-B. Sixty-seven people died. Congress introduced the ROTOR Act to mandate ADS-B on military aircraft. The House rejected it in February 2026. Military helicopters still fly without broadcasting their position. The CDA's broadcast-only model depends on universal compliance in a world where we have fatal proof that compliance is not universal, and where Congress has declined to fix it.

This is the same pattern repeating in AI agent and autonomous system design. Teams take controls that worked under human supervision, such as output review, access controls, or monitoring dashboards, and deploy them in systems where no human is watching in real time. The threat model changed the moment the human was removed.

When I first joined Prime Air, the engineering team shared an analogy. You can drive a car into a wall. The car does exactly what you tell it. But a horse will stop. A horse has its own sense of self-preservation. It sees the wall and refuses. The vision was to build a horse, not a car. A system that is independently safe.

Build the Horse, Not the Car

Step-by-step guide:

  1. Inventory every control in your autonomous or agentic system design

  2. For each control, ask: does this control depend on a human noticing something, intervening, or making a judgment call?

  3. If yes, that control is not sufficient for autonomous operation. It needs an independent verification layer.

  4. Design independent controls that do not rely on the same data source the agent uses to make decisions. For drones, this is onboard detect-and-avoid. For AI agents, this might be output validation, guardrails, or independent monitoring.

Example:

  • Instrumentation: Map each control to whether it assumes human presence (review, override, visual check)

  • Signal: If you find a control that works only because a human is in the loop, flag it. That is your single point of failure.

  • Maintenance: Re-run this audit whenever you change the level of human involvement in any workflow. Reducing human oversight changes the threat model.

Key takeaway: When you remove the human, you have to replace what the human was doing with an independent control.

Summary

Amazon pulled out of the Commercial Drone Alliance because the Alliance is pushing a broadcast-only approach to drone safety that skips onboard detect-and-avoid. From a security standpoint, that approach broadcasts the telemetry an attacker needs to hijack a drone's GPS and removes the independent safety layer that would catch the attack. We have fatal evidence that broadcast compliance is not universal, and Congress has declined to mandate it.

This is not a drone problem. It is a control design problem. The same pattern is repeating across AI agent and autonomous system architectures, where teams assume that controls built for human-supervised processes will hold up after the human is removed.

They will not.

When you remove the human, you have to replace what the human was doing. Not with assumptions. Not with policy. With an independent control that can see what the primary system cannot.

Build the horse. Not the car.

Whenever you're ready, here are 3 ways I can help:

  1. Work Together - Need a DevSecOps security program built fast? My team will design and implement security services for you, using the same methodology I used at AWS, Amazon, Disney, and SAP.

  2. DevSecOps Pro - My flagship course for security engineers and builders. 33 lessons, 16 hands-on labs, and templates for GitHub Actions, AWS, SBOMs, and more. Learn by doing and leave with working pipelines.

  3. Lunir – Fix software supply chain security vulnerabilities without the headache of manual triage and review. We fix what scanners find.

Sources and further reading

  • UT Austin Radionavigation Lab, "Unmanned Aircraft Capture and Control via GPS Spoofing" (paper). Foundational research proving covert GPS takeover requires real-time target telemetry.

  • PMC survey of GPS spoofing against aerial platforms (survey). Peer-reviewed analysis of UAV navigation vulnerabilities and spoofing techniques.

  • Insurance Journal, "Amazon Exits Commercial Drone Alliance" (article). Reporting on the CDA's counter-proposal to the FAA's detect-and-avoid mandate.

  • PMC, "Trusted Remote ID for Drone Security and Compliance" (paper). Analysis of ADS-B and Remote ID protocol vulnerabilities and cryptographic challenges.

  • FLYING Magazine, "Amazon vs. Commercial Drone Industry on Safety" (article). Reporting on Amazon Prime Air's 70,000+ flight safety record and detect-and-avoid results.

Subscribe to the Newsletter

Join other product security leaders getting deep dives delivered to their inbox for free every Tuesday.

Follow us:

Frequently Asked Questions

You have questions. We have answers.

Why did Amazon pull out of the Commercial Drone Alliance?

Amazon believes that onboard detect-and-avoid sensors are required for safe autonomous drone operations. The CDA is pushing a broadcast-only model that skips this requirement. Amazon's own data showed a near-miss with a helicopter that was not broadcasting as legally required, validating the need for independent detection.

Does this mean ADS-B is inherently insecure?

ADS-B was designed for a world where pilots are in the cockpit. In that context, it works fine. The protocol itself is not the problem. The problem is using it as the only safety layer in a system where no human can intervene. The control was never designed to carry that weight alone.

How does this apply to AI agents that don't have physical consequences?

The principle is identical. An AI agent making autonomous decisions (executing code, sending communications, accessing data) needs independent controls that do not rely on the same inputs the agent uses to make decisions. If your only safeguard is a human reviewing outputs, and you remove or reduce that review, the safeguard is gone. You need the equivalent of detect-and-avoid: an independent verification layer.

Quick Links

Supports Links

Quick Links

© 2025 Mission InfoSec. All Rights Reserved.