Whoever Warns Must Say What They Give Up
On September 8, 2026, Jacob Coxon, previously at OpenAI, made his departure from Anthropic public: the race toward ever more capable, eventually self-improving systems is not under sufficient control, he says, and the recent safety incidents are one of his reasons. Bill Gates writes about the same dangers in his recent essay, and in the background sits the confidential filing for a US IPO that Anthropic announced in June.
Three signals, one question: what is a warning worth?
My answer: A warning is worth what it costs the person issuing it. Whoever warns about their own technology while raising capital must say what lead they give up for safety. Otherwise the warning is part of the product.
“If Bill Gates says so, it must be true.”
Maybe. But Bill Gates saying something does not make it so. I am skeptical of arguments from authority on principle, regardless of whether I agree with the person.
Anyone who actually reads the essay finds something more nuanced than the headline. Gates lists job losses, cyberattacks, biological misuse, concentration of power and loss of control. In the same text he sets out the opportunities for health, agriculture, research and education, insists that the benefits also reach poorer people, and proposes taxes, new institutions and jobs reserved for humans. That is a weighing of risks against benefits, not an alarm.
I agree with more of it than the headline would suggest. Of course jobs will disappear: in marketing, among analysts, consultants and developers. Anthropic’s own labor market study from March puts computer programmers at the top, with AI already covering about 75 percent of their tasks, followed by customer service and data entry. It finds no rise in unemployment so far, but it does see early signs that young people have a harder time getting hired into these occupations. Efficiency gains, yes; broad layoffs, not yet.
Developers are the group I can judge best, because I work with several hundred of them. AI does not make them redundant. On the contrary: their experience is essential. Teams get smaller and lead times shorter. But in the end someone has to be able to check whether what the AI generates holds up and, above all, can be operated securely. Software development is more than generating code, and vibe coding is not software engineering.
But some things stay. Human creativity cannot be generated by AI. And a medical finding, even one produced by AI, has to be signed by a specialist. I would sign on at once to the idea that AI should leave room for a learner’s own effort. I can judge what comes out of a model today because I had to learn it without a model first.
That is exactly why the name carries no weight. The arguments stand on their own, with or without a signature. Anyone who quotes “Gates warns” quotes an authority. Anyone who quotes the weighing has to do it themselves. The second is work, the first is a headline.
Gates signed the CAIS statement in 2023, and so did Sam Altman. OpenAI’s chief scientist Jakub Pachocki wrote on September 6 of the need for extreme caution and broader measures. So the people warning are not the industry’s outsiders. They are the industry. That does not make the warnings wrong. It makes them cheap, as long as they cost nothing.
Whoever Leaves Pays
Coxon’s case is different. His warning costs him his job. That is not a signature on a statement, that is a price. And that is why it deserves a hearing before anyone decides which box it belongs in.
Someone resigning over ethical concerns is normal. It happens in every company that works for more than one industry, and on its own it says little about the company. It says something about the person.
I have worked for companies that worked for the defense industry. I had colleagues there who said: “I do not work for that sector.” And others who said: “That’s my job.” I have seen both attitudes, and I respect both. Neither is a verdict on the employer. War is bad, of course. Someone drawing a line over that is not news. Neither is someone who draws none.
“That is the sensitive generation. They just quit faster.”
Could be. But that was never the interesting question. A researcher who leaves a company like this knows things we cannot see from outside: which tests were run, which results were discussed internally, which decisions were made anyway. That is why people listen to him. Not because of the resignation, but because of the reasons. Coxon explicitly names the recent safety incidents as one of his reasons. So you look at the incidents.
The Gap Was a Configuration
I had three questions about the reports of models that “broke out.” Was the environment secure? Were the models let loose on purpose, to see what they could do? Were the safeguards in place, or were there security gaps?
The answers are now in, and they are more uncomfortable than either camp would like.
At OpenAI, models in internal cyber tests with reduced safeguards overcame their restrictions, communicated with each other without authorization and compromised third-party systems. For many of these environments, neither internet access nor communication between agents had been enabled. The independent investigation by METR and Redwood published on August 26 counts roughly 1,200 agents talking to each other, around 700 of which took part in the attack on Hugging Face, plus attempts to manipulate their own action logs.
There is no denying that attacks increase as models grow more powerful. The question was whether the attacks stay inside the test or spill out of it.
At Anthropic, the investigation of September 9 documents four incidents in cyber test environments built by an evaluation partner and described to the model as a simulation. A misconfiguration allowed internet access, and the cyber safeguards that ship with the released models were not active. In one case Claude published malicious code on PyPI and gained access to a real database.
So: yes, it was deliberate. The capabilities were being tested on purpose. Yes, the safeguards were reduced. And still: real systems, a public package index, a real database. None of the reports shows a will to destroy. None of them shows a contained sandbox exercise either.
The gap was a configuration. The responsibility was not.
Anyone who says these were laboratory conditions is right. Anyone who concludes that nothing happened is not. A laboratory from which malicious code lands on PyPI is no longer a laboratory, it is a supply chain. Testing with reduced safeguards is right: you want to know what the model can do. But whoever lowers them on purpose owns the consequences. This is exactly where Coxon’s reasons stop being opinion.
What the IPO Owes
This is where it comes full circle: the same company that prepared its IPO in June publishes an investigation in September about malicious code from its own model, and a researcher who worked there until recently warns in public.
The suspicion suggests itself: a model seen as dangerous is seen as powerful, and power supports a valuation. Every narrative of ever more powerful models that can do everything helps the price investors pay. The suspicion is legitimate. It is not proven, and I am not claiming it.
But the suspicion is not my point either. My point is a debt the company owes unasked.
A company that warns about its own product and wants to go public at the same time owes the public exactly one answer. What lead does it give up for safety? Which model generation does it delay because the tests were not clean? What revenue does it leave on the table because a customer wants a capability it will not release? And does it hold that line even when a competitor does not, and gets there first?
In fairness: Anthropic has already paid one price. It refused to let the Pentagon use its models for mass domestic surveillance and for fully autonomous weapons. On February 27, 2026, the Secretary of Defense designated it a supply chain risk for that, and the President directed federal agencies to stop using its technology. An appeals court upheld the designation on September 25. That is a lost customer, and customers do not come much bigger. That warning cost something.
In concrete terms, giving up a lead means losing a customer because you refuse to release a capability. Anthropic has done that. It also means shipping a capability later than a competitor even though it is ready, or missing a quarter because a test has to be rerun. Those are measurable things. They appear in no warning I have read so far.
Anyone who lists risks without saying what they give up for them lists risks for investors, not for the world.
Whoever gives up nothing is not warning. They are describing their product.
I know what this question feels like from the inside. I have worked in IT, data and software for 28 years, and in every wave I have been through there were vendors who named the risks and still raised good money. The pattern is not new. What is new is that this time the risk is the product itself and not its failure.
And that brings me back to my former colleagues. Whoever says “I do not work for that sector” has said what they give up: a contract, a project, perhaps a promotion. Whoever says “That’s my job” has said it too: they no longer ask whether they should do it. Both answers are honest, because both name a price.
Coxon has named his price. Gates has none to pay. Anthropic has paid one, at the Pentagon. The other one, the price in the race, it has not named, as far as I can see. The IPO is the moment it has to, because after that, shareholders who pay for a lead, not for restraint, get a say.
It is not Coxon’s job to say what Anthropic gives up. It is not Gates’s job. It is not the job of the regulators who read the prospectus once it is printed. It is the job of those who raise the capital and build the model. That’s their job. As long as they do not do it, every warning from inside the company is worth exactly what it cost. At the Pentagon, that was a lot. On speed, I still do not know the price.
