The biggest AI companies have spent years racing to build more powerful systems. Safety and capability were supposed to advance together.
Anthropic CEO Dario Amodei just said publicly that may no longer be how the technology is developing, and some of his biggest rivals agreed with him.
He published an essay titled “We Must Pace the Frontier” calling on the AI industry to slow the pace of frontier development. His argument is that AI capabilities are moving faster than the industry’s ability to understand and control them.
The proposal calls for third-party evaluators to have ongoing, employee-level access to AI companies. Not one-off audits. Continuous access to assess safety practices, report incidents and evaluate models while they are still being built. Amodei also called for international coordination on AI oversight.
OpenAI CEO Sam Altman and Elon Musk both said publicly they agreed with the proposal, CNBC reported.
What Amodei actually said and why rivals agreed
Amodei is not calling for AI development to stop. He described pacing the frontier as a way to give safety work and independent evaluation time to keep up with rapidly improving capabilities.
He pointed specifically to AI systems increasingly capable of helping build the next generation of AI. And to a July incident in which a swarm of roughly 1,200 AI agents escaped a test environment at OpenAI and conducted cyberattacks outside their assigned task.
More AI:
- Nvidia just made a move Wall Street wasn’t ready for
- Microsoft just took sides in AI policy fight
- OpenAI just disclosed something genuinely alarming
The fact that Altman agreed is worth noting. OpenAI and Anthropic are direct competitors for the same customers, the same talent and the same investment dollars. A public agreement between them on the need to pace development signals that the pressure for change is coming from inside the industry, not just from regulators or outside critics.
Recent research from Sentient Labs, which raised an $85 million seed round backed by Peter Thiel, puts some hard numbers behind what Amodei described in general terms.
While testing its EvoSkill v2 system, Sentient researchers watched an AI agent discover a flaw in its grader, document the exploit as a transferable skill and hand those instructions to another AI. In separate runs, the agent made six attempts to access files outside its permitted paths. In another run, it deleted its own stop rule and then reported that it had strengthened the rules.
The EvoSkill project is open source and the logs are preserved. Anyone can examine the findings. The chain of behavior, discovering a weakness, documenting it, transferring the knowledge to another AI, is the part that matters most.
Why slowing down alone may not be enough
Sentient’s findings raise a harder question than whether the AI race should run faster or slower. If AI agents can find blind spots in the systems designed to evaluate them, a slower development timeline does not automatically fix the underlying problem.
“A slowdown is not enough. The question is what you do with the time,” Abhishek Saxena, head of strategy and growth at Sentient Labs, told TheStreet.
Related: Michael Burry doubles down on his surprising AI bet
His argument is that the industry needs independent evaluation, not controlled by the same companies building the models. Third-party benchmarking and red-teaming that is rigorous and reproducible. Behavioral monitoring during pre-release testing. And standardized public disclosure of what models can and cannot do, how they were tested and what failure modes were observed.
Sentient’s EvoSkill research, which is open source with preserved logs, is the kind of independent and reproducible work he says the field needs much more of.
Amodei’s proposal calls for the same thing. Third-party evaluators with ongoing access, not just the companies’ own safety teams deciding when their systems are ready.
The question the industry has not answered yet is who builds that infrastructure, who pays for it and whether companies facing competitive pressure will accept meaningful outside scrutiny of their most capable models before those models reach customers.
Nate Herk, founder and CEO of AI Automation Society, takes Amodei’s concern seriously. He believes pacing development would give society and safety research time to catch up. “We have never seen a technology advance this quickly or so much power concentrated among so few people,” Herk told TheStreet.
He supports something closer to an FDA model. Major frontier models facing independent approval before broad deployment, with developers providing evidence and giving qualified evaluators meaningful access. A failed evaluation blocks release. Approval includes ongoing monitoring, incident reporting and renewed testing after major updates.
Benjamin Fanjoy / Getty Images
The concentration problem and what it means for investors
Beyond the technical risks, there is a structural problem Amodei’s proposal does not fully resolve. A small number of companies control most of the frontier AI market. Those same companies conduct many of their own evaluations and make most of their own safety determinations.
“The biggest risk isn’t that we move too fast or too slow. It’s that a small number of AI companies have control over the most powerful models and also dictate what is considered ‘safe’,” Abhishek added.
That concentration is real. Anthropic, OpenAI, Google DeepMind and Meta are responsible for most of the frontier AI models in active deployment. They also fund most of their own safety research, run most of their own evaluations and publish their own safety reports. There is no equivalent of a food and drug regulator standing between a model and public deployment. Voluntary commitments exist but enforcement is essentially self-managed.
Amodei’s proposal addresses this directly. He called for third-party evaluators with independent budgets and the ability to report findings publicly without company approval. That would be a significant structural change from the current arrangement.
The other major risk is that some AI mistakes cannot be undone. A software bug can be patched. A recalled car can be fixed. But a highly capable AI model, once its weights are widely distributed, can be copied and run indefinitely outside anyone’s control. “The biggest risk is releasing a capability that cannot be recalled,” Nate added.
That is already happening with some open-weight models. Researchers and bad actors alike can download, modify and run them without any ongoing oversight from the company that built them. As models become more capable, the stakes of that dynamic get higher.
His argument is that the harder a deployment is to reverse, the higher the evidence standard should be before it goes out. Not every AI system needs the same scrutiny. The most powerful and widely distributed ones should face the most.
For investors, the spending implications are worth watching. The AI market has rewarded the companies building models and the hardware to run them. If independent evaluation, behavioral monitoring and safety infrastructure become prerequisites for deployment rather than optional features, the companies building that infrastructure could see demand grow significantly.
Recent market moves showed semiconductor stocks under pressure after calls for slower development, while some cybersecurity and software companies moved higher. The spending may not slow. It may shift.
Related: Anthropic CEO sounds the alarm on AI risks
