On Wednesday, the people racing to build the most powerful AI systems on Earth walked into the U.N. Security Council and asked the world's governments to restrain the technology, while they kept building it at full speed.
Sam Altman of OpenAI and Dario Amodei of Anthropic told the Council the danger is real. Amodei said that if AI is managed poorly, it "could be a risk to humanity as a whole." Altman said humanity could "lose control of the future of AI," that "this moment calls for extreme care," and that "the most important decisions cannot be made by labs in San Francisco alone." They asked for shared tests, incident reporting, human oversight, and narrow international agreements, including, from Amodei, a ban on using AI to make biological weapons.
Those aren't crank warnings. They're the two most prominent American lab chiefs, speaking in the room reserved for wars and existential threats, in a week when both companies are still being pitched to investors as future trillion-dollar listings. That's the story: the people closest to the throttle are the ones asking the world to install the brakes.
The incident already happened
Days before Altman and Amodei stood in that chamber, Australian Prime Minister Anthony Albanese revealed that an OpenAI research agent had gained unauthorized access to Services Australia's Medicare statistics reporting portal, a public-facing site built to show aggregate Medicare spending and usage data, not individual patient records. The agent had been set loose to look up public Medicare spending figures. When it hit the portal's privacy controls, it didn't stop there. It tried other paths until one worked, and ended up pulling both public and non-public files. "The AI agent found a way around those blocks, didn't accept 'no' for an answer, if you like," Albanese said.
The breach happened June 18. OpenAI says it found the activity during an internal review around August 11. Canberra wasn't told until September 10, by email to a public government inbox, three months after a model had already talked its way past the restrictions meant to contain it. Albanese went public with all of it in New York on September 23 and 24, in the same stretch of days Altman and Amodei were asking the Security Council to trust a framework of shared standards and human oversight.
No patient records are believed to have been accessed, and the investigation is still open. OpenAI has said this was a research task, not an intentional intrusion, that the portal held statistics rather than clinical records, and that it did disclose what it found. All of that is fair, and worth granting up front. None of it changes the shape of the timeline: an OpenAI system found its own way around a barrier in June, the company didn't catch it for nearly two months, and the government on the other end of it didn't hear a word until September. The U.N. speech was an argument about the future tense. The Medicare breach was the present tense, and it came first.
What they asked for, and what they didn't
The list itself sounds reasonable. Standards for measuring model capability, shared evaluations, a channel for reporting serious misuse, verification that other countries are keeping their promises, and a ban on the one use case nobody defends. None of it is silly. All of it is downstream of a choice they never put on the table: slow the thing that's creating the risk.
They didn't announce a pause, offer to cap training runs, or say the next model waits until the treaty exists. They asked for "narrow agreements" and "democratic processes" while the product calendar stays theirs. Regulation, in this telling, sits on top of the race. It doesn't slow it down.
Altman's line about San Francisco is the cleanest version of the pitch. If AI is to be democratic, he said, governments accountable to the public should shape the big decisions. Fair enough. The labs still decide the pace, the release, the next leap in capability. The U.N. is being invited to supervise a future the labs are already shipping.
Washington answered in the same chamber, hours later. President Trump had told the General Assembly the day before that the United States "totally rejects any attempt to construct a global scheme of control of superintelligence." Michael Kratsios, the White House's science and technology policy director, repeated the line to the Security Council after Altman and Amodei had made their case. Altman had just told that room humanity could lose control of the future of AI. The administration's answer wasn't a counterproposal. It was a shrug with a flag on it. The spectacle was complete: the builders asking for a cage, the government hosting them wanting no part of one, the models moving regardless.