Get you up to speed: Anthropic urges AI labs to pause, warns humans risk losing control

What remains unclear — It is not specified how Anthropic plans to verify adherence to the proposed coordinated pause among AI labs.

Anthropic calls for AI labs to consider a pause amid control concerns

Economy|TechnologyAnthropic urges AI labs to pause, warns humans risk losing control

It warned that rapid advances in technology could soon allow AI systems to improve themselves faster than society can control the risks.

Anthropic is calling on major artificial intelligence labs to consider a coordinated and verifiable pause in development [File: Patrick Sison/AP Photo]

Anthropic is proposing that the world’s top artificial intelligence companies come up with a coordinated way to pause development of advanced AI systems, warning that the technology is improving so quickly that there’s a risk humans would lose control.

The company behind the Claude chatbot said in a blog post on Thursday that, as cutting-edge AI gets increasingly faster at carrying out tasks, “it would be good for the world to have the option to slow or temporarily pause” its development.

Anthropic said its internal research institute plans to explore the issue in collaboration with others and “take actions” to help build the systems for a credible slowdown or pause, without being more specific.

Anthropic rival OpenAI argued for a different approach in a report published on Wednesday, saying that “democratic governments — not private companies acting alone — must ultimately determine the rules, safeguards, and accountability mechanisms”.

“Our view is that decisions about the pace of AI innovation should not be left to any one lab, company, or special interest group,” it said.

AI models are getting faster, with rapid increases in how quickly they can carry out software tasks like coding on their own, Anthropic said in its post. Based on current trends and given enough computing power, an AI system could be able to design and develop its own successor, in what is known as “recursive self-improvement”.

Self-building AI would be a major technological milestone that would bring benefits in science, healthcare and other areas, Anthropic said, but it “also might increase the risks of humans losing control over AI systems”.

Some tech industry figures have long warned of such a scenario.

Anthropic’s post comes after a different warning this week from a team of researchers at the University of Toronto who showed how AI tools could be used to create a new kind of AI “worm” that adapts its hacking strategy as it spreads from device to device and takes over a vast computing network.

“I think it’s really important that people understand that it’s not just the biggest, most powerful language models that pose the security concerns,” lead researcher Nicolas Papernot said in an interview.

The authors of the Anthropic post, company cofounder Jack Clark and Marina Favaro, head of its research institute, said the pause would be used to enable “societal structures and alignment research” to keep up with AI advances. Alignment is industry shorthand for making sure the technology matches human values and intentions.

The proposed coordination would let advanced AI labs verify that global rivals have actually stopped or slowed their work, “and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret”.

The company said a coordinated global mechanism is needed because, without it, a slowdown in AI development could let the “least cautious” players catch up and add to pressure on companies and governments as they make tough choices about AI safety.

Fears that advanced AI systems may get out of human control and cause societal harm have risen as the technology becomes increasingly capable. Anthropic’s own Mythos model sent shockwaves through industries, including banking and software, earlier this year with its ability to find vulnerabilities in existing code.

But regulation has been slow, especially in the US, where most leading AI labs are based. A Trump administration executive order earlier this week put the onus on the labs themselves, asking them to voluntarily submit their most capable models for government cybersecurity testing before public release.

Safety focus

AI researchers have also urged a pause before, but have had little success. Elon Musk, who owns AI lab xAI, was among the backers of a 2023 push by the non-profit Future of Life Institute to halt AI development for six months to allow time for safety guardrails.

Anthropic has long positioned itself as a safety-focused AI lab. Earlier this year, it refused to let the US military use its models for domestic surveillance and fully autonomous weapons, prompting backlash from the government, which put it on a national security blacklist, set to take effect later in 2026.

Anthropic’s post comes as the company and ChatGPT-maker OpenAI race to sell shares on the stock market, in an IPO that could value Anthropic at nearly a trillion dollars.

Papernot notified Canadian cybersecurity authorities prior to releasing his report, which shows how researchers developed the worm in a laboratory by using an “open-source” AI tool that is easy for software developers to cheaply access and modify.

“In the past, cyber attackers would focus on targets that are very high value,” he said. “Banking systems, hospitals, electricity grids, water treatment systems, schools.”

Papernot agreed that there should be more collaboration between companies, government agencies and academic researchers to develop countermeasures as AI-powered hacking tools supercharge the search for computer vulnerabilities.

“That old laptop you have in your basement that you don’t check on regularly doesn’t seem like a very high-value target, but it can be used as a launch pad to attack these higher-value targets,” he said. “Anything connected to the internet is now at risk because of how low the cost has become to mount these cyberattacks.”