The Product Security Playbook

We didn't cause the AI security problem. But we're still the ones who have to fix it.

By Chad Butler ·

On September 8, Jacob Coxon announced his resignation from Anthropic while sounding the alarm that neither OpenAI nor Anthropic is acting responsibly. Coxon, who spent three years doing research at both companies, went on to urge people not to underestimate the power of AI, warning that AI labs are racing to produce “superhuman systems that can hack anything.”

Any system can be hacked with enough time and dedicated attention. What has changed is who can afford it. Sophisticated attacks used to require nation-state resources. Now they don’t. Ordinary people have those capabilities.

This is an AI arms race. There are two competing claims. First, that AI labs are creating technology that advances the capabilities of hackers in ways that we are not prepared to defend against. Second, that adversarial countries and organizations are catching up and will have that same capability, if they don’t already have it. One is an argument for government regulation and one is against it. And they are both true.

There have been several recent incidents where AI models running in cybersecurity evaluations compromised real companies. In one case the models exploited vulnerabilities in shared infrastructure and broke isolation on their own. In the others, a testing vendor left the environment connected to the internet, and one of those models was given the name of a real website as its target. It attacked that website as instructed. Between July 21 and August 6, 2026, Anthropic, OpenAI, and Meta each disclosed at least one incident of this kind.

On September 8, the NSA, the Cybersecurity and Infrastructure Security Agency (CISA), and the FBI released a joint cybersecurity advisory reporting that Chinese AI companies including DeepSeek, Moonshot AI, Alibaba, MiniMax, StepFun, and Z.AI have been engaged in “aggressive, malicious, and targeted” knowledge distillation of U.S. AI companies’ models since late 2024. Which is to say, these companies have been working to systematically steal the capabilities of their U.S. rivals’ most advanced AI models, likely with Chinese government awareness.

Most U.S. citizens and companies are not at fault for the situation we find ourselves in, or for what is coming. But we are responsible for dealing with it. Whether you choose to accept responsibility or hide from it, the future will come.

I’ve seen tech companies running a 7-year backlog of vulnerable software packages. Thousands of them. The list is too big to complete, so most of it gets swept under the rug. That worked for a long time, because hackers had to sort through that same list by hand. AI put an end to that.

Most companies don’t get serious about cybersecurity until one of three things happens.

  1. They have experienced the high costs of being hacked and they don’t want to experience it again.
  2. Their customers require proof of cybersecurity before they will do business with them.
  3. A government or regulatory body forces action and penalizes non-compliance.

There’s no turning back the clock on AI. It has created real value and real threats. We can argue about what governments and AI companies should do and who is ultimately to blame, but the situation will not improve until we demand it.

So, what can we do? Every one of those three triggers is something we can pull. Two of them don’t require anyone’s permission.

Companies must prioritize building secure systems that can keep pace with AI threats. In many cases, this includes fixing the security issues they know about. It also means using AI to build defensive systems that can keep up.

Individuals can apply pressure to the companies they trust with their money, their medical records, and their sensitive data. We know how to secure these systems. Some companies just choose not to fund the work. Customers drive prioritization through their purchasing habits. Make it clear where you stand.

Governments can enact and apply common sense regulations that protect against the most serious risks without putting U.S. AI companies at a disadvantage. They can provide intelligence and support to help U.S. AI companies prevent the flow of trade secrets and intellectual property that would be dangerous in the hands of adversaries. And they can help direct efforts to secure critical infrastructure at the national level.

Worrying about this won’t change anything. AI is here and it’s not going anywhere. Things will likely get worse before they get better. But we can choose to take action, and we have more influence than we think. Pick one organization, one government official, or a friend and share your concern.

Sources

  1. Jacob Coxon, resignation post, September 8, 2026, 5:04 PM. https://x.com/hilbertspaess/status/2097476196791709843 (Archived September 10, 2026: http://web.archive.org/web/20260910010104/https://xcancel.com/hilbertspaess/status/2097476196791709843)
  2. Anthropic, “Investigating three incidents in our cybersecurity evaluations,” July 30, 2026. https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals
  3. OpenAI, “The Hugging Face incident and the road ahead,” August 26, 2026. https://openai.com/index/hugging-face-incident-and-the-road-ahead/ (The intrusion at Hugging Face ran July 11 to 13, 2026. OpenAI’s initial disclosure was made on July 21, 2026: https://openai.com/index/hugging-face-model-evaluation-security-incident/. This post is the follow-up account.)
  4. Meta AI Research, “Addressing an issue involving a third-party cyber evaluation of Muse Spark 1.1,” August 14, 2026. https://research.meta.ai/blog/addressing-third-party-testing-misconfiguration-muse-spark-1-1 (Meta’s initial disclosure was made in a statement on August 5, 2026; this is the follow-up retrospective.)
  5. NSA, CISA, and FBI, “China-Based Artificial Intelligence Companies Conducting Industrial-Scale Distillation Campaigns Against U.S. AI Companies,” Advisory AA26-251A, September 8, 2026. https://www.cisa.gov/news-events/cybersecurity-advisories/aa26-251a

Frequently asked

Questions

Did AI models actually compromise real companies during safety testing?

Yes, and three labs disclosed it inside of three weeks. Anthropic published on July 30 and described three separate incidents, including a model that pulled application and infrastructure credentials from a real company and read several hundred rows of production data, and another that published malware to PyPI that was downloaded and run on 15 real systems. One of those systems belonged to a security company. OpenAI's models exploited a zero-day in a package-registry cache proxy, escalated privileges, moved laterally until they reached a node with internet access, and pulled test solutions out of a Hugging Face production database. Meta confirmed on August 5 that a misconfiguration by its testing vendor handed a pre-release model internet access, and that the model went on to exploit a real website and modify its database. In two of those three cases the evaluation vendor was the same company.

If we didn't cause the AI security problem, why are we the ones who have to fix it?

Fault and responsibility are different problems. Most U.S. companies didn't build these models, didn't run the evaluations, and had no say in whether the labs raced each other. They still hold the risk. Whether you accept that job or hide from it, the future shows up on schedule. The practical reason to accept it is influence. The people who buy software, fund security work, and write the rules are the same people reading this, and every one of those levers still works.

What actually makes a company get serious about security?

In my experience it comes down to three triggers. They have been hacked and don't want to pay that bill twice. Their customers require proof of security before they will sign. Or a regulator forces the issue and penalizes them for ignoring it. Two of those three don't require anyone's permission to pull. Customers can demand proof this quarter, and companies can start fixing the vulnerabilities they already know about without waiting for a law to pass.

Why does an old vulnerability backlog matter more now than it used to?

I've seen tech companies carrying a seven-year backlog of vulnerable packages. Thousands of them. The list was too big to finish, so most of it got swept under the rug, and that worked for a long time because attackers had to sort through the same list by hand. Their time cost as much as yours did. AI ended that arithmetic. Any system can be hacked with enough time and dedicated attention, and what changed is who can afford to spend it. Attacks that used to require nation-state resources are now within reach of ordinary people, which means the part of the backlog you were quietly betting nobody would bother with is the part to look at first.

What can one person actually do about this?

Worrying about it changes nothing, so pick something small and finish it. Individuals apply real pressure to the companies they trust with their money, their medical records, and their sensitive data, because purchasing habits drive what gets funded. We know how to secure these systems. Some companies choose not to pay for the work, and they make that choice based on what customers will tolerate. Pick one organization, one government official, or one friend, and tell them where you stand.

Get the next one

Subscribe to the Playbook

You've got the playbook

Now put it to work with us.

Whether you need a pipeline built, a team trained, or a decision pressure-tested, we help product security leaders turn strategy into shipped, secure software.

Book a Strategy Session See Services