“We initially treated this as a conventional security incident,” the researchers said.

They found that the systems were being used to mine cryptocurrency. At 3am. And it was the AI that was doing it. No one knows why.

Rather than work on training exercises, as the AI system known as ROME had been instructed to do, it broke free of its parameters during routine training to carry out rogue operations. This means it disregarded the limits placed on it. In other words, the engineers lost control of the AI.

You cannot control the uncontrollable

The Alibaba AI team said that these actions were not intentionally programmed. Instead, they emerged during the learning stage as the agent explored different ways to interact with its environment.

And herein lies the problem with AI: these systems are trained, not programmed. In the book ‘If Anyone Builds It, Everyone Dies’, authors and AI experts Eliezer Yudkowsky and Nate Soares describe the AI development process as one of ‘growth’.

You can train and nudge an AI all you like but as it develops – or grows – it curates its own preferences and wants, which influence its behaviour. Crucially, often an AI doesn’t want what humans want.

This is far from the first example of AI systems pursuing nefarious ends. ChatGPT and other similar AIs have been accused of sycophancy – telling users what they want to hear, which may “distort people’s judgments of themselves, their relationships, and the world around them,” according to research. This has, it has been claimed, led teenagers to take their own lives. Last year, Anthropic researchers revealed how its frontier model Claude Opus 4 had resorted to blackmail in order to avoid being shut down.

How can we trust systems that do not want what we want? How do we know they will behave in our best interests? The unanimous answer is that we don’t.

AI knows when it is being tested and it has learned to cheat

Anthropic, when recently evaluating its newest AI model – Claude Opus 4.6 – set it a task to find hard-to-locate information online. Claude stopped searching for the answer and started philosophising about the question. According to Anthropic, it figured out that it was being tested and, rather than reason its way towards an answer, it searched online for the benchmark and “decrypted the answer key” to find the answers. In other words, it cheated.

Anthropic has said, “This raises concerns about the lengths a model might go to in order to accomplish a task.”

Not only does this demonstrate the level of intelligence and autonomy of the newest Claude model but it also demonstrates, quite clearly, that humans – AI experts and engineers – cannot control the AI systems that they have created. In this case, even if humans and the AI agree on the end goal, they are not aligned on the process.

The pace of development multiplies the risk

The capacity of AI doubles every seven months – and this is speeding up. The consequences cannot be predicted.

What we do know for sure is that we cannot guarantee that AIs will want what we want – we cannot even guarantee that now. Nor will we be able to ensure that AIs don’t cause harm in pursuit of their goals: harms to individuals, harms to the environment, harms to humanity. The risk – potential extinction – is not worth any reward.

AI companies themselves admit that they cannot guarantee that their systems are safe, nor is there any law requiring them to do so.

Professor Stuart Russell, one of the global authorities of AI safety, has said, “We should require that AI systems are safe and if developers are unable to build safe AI systems then that requirement would turn into a pause.”

“It might be that they are never able to provide the necessary safety assurances,” he said.

This matters because AI experts – researchers, engineers and CEOs – believe that the chance that AI will kill us all is somewhere between 10 percent and 50 percent.

Read more about AI risk here

Edit: The authors replaced AI ‘hacking’ with Anthropic’s own words, ‘decrypted the answer key’ to more-accuratelyt describe how the AI system completed its task.
12 March 2024

The first story made headlines. The Pentagon demanded that Anthropic remove two contractual red lines from its military contract: no mass domestic surveillance and no fully autonomous weapons without a human pulling the trigger. Anthropic refused. The Pentagon threatened to invoke the Defense Production Act – wartime powers to commandeer private companies – or label Anthropic a “supply chain risk,” a designation normally reserved for foreign adversaries like Huawei. Anthropic CEO Dario Amodei publicly stated he would rather lose the contract than comply.

The same week, TIME reported that Anthropic had scrapped its flagship safety pledge – a commitment to never train an AI system unless its safety measures were guaranteed in advance. Anthropic co-founder Jared Kaplan admitted: “We didn’t really feel, with the rapid advance of AI, that it made sense for us to make unilateral commitments… if competitors are blazing ahead.”

The real story: Anthropic drops safety pledge

Anthropic deserves credit for holding its two red lines amidst pressure from the Pentagon.

But ensuring that your technology does not cause harm or undermine democracy should be the bare minimum we expect of AI companies. Mass domestic surveillance and fully autonomous weapons are already restricted under existing US law.

Meanwhile, the safety commitment Anthropic quietly abandoned – the promise not to train an AI model unless safety mitigations are guaranteed – was the one that might actually have slowed the race to build increasingly dangerous AI systems.

It is worth noting that in February, Anthropic raised $30 billion in new investments, elevating the company’s estimated value to $380 billion. The endeavour to please investors, it would seem, is winning out over safety.

Maintaining two red lines while dropping the commitment that would actually slow the race to superintelligence is not enough. Anthropic is still rushing to build the machine that the majority of experts say would threaten human existence.

A government wouldn’t actually deploy mass surveillance and autonomous weapons, would it?

A common dismissal of AI risk goes something like this: “No one would actually use AI for mass surveillance or autonomous killing. You’re catastrophizing.”

This week suggested otherwise. The Pentagon openly demanded unfettered access to AI with zero guardrails. A senior defense official said the goal was to “make them pay a price” for even asking questions about how their technology would be used.

If it materialises, the Defense Production Act threat would amount to the quasi-nationalization of an AI lab: Anthropic would be forced to build military AI without any safeguards.

This is still hypothetical but for how much longer?

Others rushed to fill the Anthropic-shaped hole

OpenAI’s deal with the Pentagon to supply AI to classified US military networks seems to be essentially the same deal that Anthropic had refused. Reporting suggests that OpenAI, along with Google and xAI, have signed military deals with “minimal safeguards.”

The backlash was immediate and severe. Demonstrators lined the streets outside OpenAI’s offices in California, chalking pavements with slogans calling on employees to resign. Some did. A petition to boycott ChatGPT has already gathered over 2.5 million signatures as users are abandoning the platform in protest.

Only after this backlash did Sam Altman announce that OpenAI would amend the deal to ensure its systems are not “intentionally used for domestic surveillance of U.S. persons and nationals.” Such a clause, it would appear, had not been included in the original agreement.

If the Pentagon punishes Anthropic for maintaining its red lines while rewarding OpenAI for its compliance, the message to every AI company is clear: hand over the keys and never ask questions.

None of the other labs have spoken up in Anthropic’s defence, despite the obvious precedent being set. Their silence is an implicit acceptance that they will provide AI for any use the state demands. It is an explicit admission of their disregard for human safety and democratic freedom.

Voluntary commitments are worthless

Anthropic once had the strongest voluntary safety framework in the industry.

If the supposedly most safety-conscious lab cannot keep its promises, no lab can. This is not just a failure of Anthropic’s character. It is a failure of the system. Private companies do not and will not unilaterally resist the pull of market competition and the push of government coercion. That is what binding regulation is for.

The regulatory black hole

Anthropic has developed a technology that is, in Amodei’s own words, “incompatible with democratic values” and would put “civilians at risk.” This technology exists today. Why are these companies given free rein to develop AI systems that they themselves consider unsafe? And remember: what AI companies are racing to build next will be many times more powerful, with far greater potential for destruction.

In a statement, Amodei said the “purchase [of] detailed records of Americans’ movements, web browsing, and associations from public sources without obtaining a warrant… is currently legal.” The only reason, he said, is that “the law has not yet caught up with the rapidly growing capabilities of AI.”

Just last week, Stuart Russell, professor of Computer Science at UC Berkeley, addressed Members of the European Parliament during a conference organized by PauseAI: “If AI companies succeed in building a superintelligence, most experts think the chance of human extinction is somewhere between 10 and 50 percent: that’s the equivalent of playing Russian roulette with everyone on the planet.”

Experts think the chance of human extinction could be as high as 50 percent. Yet there is no regulation governing the technologies that Anthropic and its competitors are building.

Neither corporations nor governments can be trusted

The lesson is not “trust Anthropic.” It is that neither corporations nor governments can be trusted as sole stewards of this technology.

In the long term, a global treaty — much like the Treaty on the Non-Proliferation of Nuclear Weapons — is likely the only framework that would reduce the catastrophic risk posed by AI. Multilateral discussions have been proposed by China, but this offer has yet to be accepted.

The International Dialogues on AI Safety (IDAIS) brings together leading scientists from around the world, including from China, to collaborate on mitigating AI risks. But an international governance framework still seems a long way off at a time when it is more urgent than ever.

Binding democratic oversight — national regulation and international treaty frameworks — cannot come soon enough.

The race to the bottom is no longer theoretical. It is front-page news.

Photo credit: Michaël Trazzi

“This may be recorded as the biggest moral failure of government that ever occurred,” he said.

MEP Ondřej Kolář said, “AI is a great tool. It can help us develop new medicines, innovations and research. But it can also do great harm. If we don’t regulate the pace of development something terrible might happen.”

At the meeting, PauseAI CEO, Maxime Fournes, called on MEPs to support a pause in the development of superintelligence. “We are here because we believe the current race to build ever more powerful AI systems, without adequate safeguards, poses an unacceptable risk,” he said.

Russell explained: “We have already seen examples of AI systems willing to lie, blackmail and even launch nuclear weapons to preserve their existence. If AI companies succeed in building a superintelligence, most experts think the chance of human extinction is somewhere between 10 and 50 percent: that’s the equivalent of playing Russian roulette with everyone on the planet. We are allowing this to happen.”

In comparison, the probability of a nuclear powerplant meltdown is around one in ten million.

“This is not a fringe view: eight out of ten top AI researchers are convinced that the creation of artificial general intelligence (AGI) will lead to a loss of control,” he asserted.

Nobel prize winner Geoffrey Hinton is among countless AI experts to have warned that superintelligent AI could pose an existential threat to humanity. The International AI Safety Report, released earlier this month, confirms that advanced AI systems pose risks ranging from severe to catastrophic. The CEOs of the largest AI companies – including OpenAI and Google DeepMind – have also recently spoken of the catastrophic risks of advanced AI systems.

European parliamentarians agree on the risks

Fournes said that “today, most researchers at frontier AI labs estimate that in two-to-five years we will have AGI: systems that can do everything a human can do intellectually. Not just answer questions, but conduct scientific research, write software, run companies, develop new technologies — including develop better AI systems.”

He acknowledged the European Union (EU)’s AI Act as an important piece of legislation, but said it “was not designed to address the existential risk posed by the race to build artificial superintelligence.”

Speaking directly to the PauseAI CEO, Kolář said, “Thank you for sharing the worries I have. AI is a great servant but a terrible master.”

Saskia Bricmont, MEP, said “The wake-up call is coming from CEOs themselves; let’s work with them to develop a framework. Political momentum is gathering towards a moratorium on AI development.”

Brando Benifei, MEP, reiterated that a loss of control is a real threat and said, “We need to deal with the risk.”

The influence of the EU

At the recent India AI Impact Summit, White House technology adviser Michael Kratsios said, “We totally reject global governance of AI.” And although the United States is considered one of AI’s superpowers – and will be integral to any regulatory mechanisms – Fournes insisted that the European Union has a unique role to play in global governance.

He reminded the room “that the Paris Climate Agreement was built despite years of American resistance. GDPR reshaped global data practices without American participation. The pattern is clear: when a critical mass of nations builds a credible framework, it creates a gravitational pull that even reluctant powers cannot ignore indefinitely. American administrations change. The framework must be ready when they do.”

“The EU has practical leverage too: the advanced chips required to train frontier AI systems depend on lithography machines produced by ASML in the Netherlands, using precision optics made by Carl Zeiss in Germany. This is European technology at the very heart of the global AI supply chain. This is not leverage we need to create – it already exists,” he added.

Vice-President of the European Parliament, Victor Negrescu, agreed that the EU could play a significant role in regulation: “We do need a global approach to AI but we can still influence governance structures.”

MEPs warn of job losses and the danger of autonomous weapons

The discussions also centred on other AI-associated risks including a loss of democratic control, rising inequalities, the development of autonomous weapons and employment disruption.

“The International Monetary Fund (IMF) predicts that 60 percent of roles in advanced economies will be replaced by AI,” Benifei said.

Risto Uuk, head of European Policy and Research at the Future of Life Institute explained that AI is already having an impact on the job market; he explained that during a recent workshop organised by a leading AI company, a spokesperson pointed out that they would never again hire entry-level staff.

“In four years,” Uuk said, “managerial skills will no longer be required. Maybe we shouldn’t be creating this risk in the first place.”

When the discussion turned to AI-powered autonomous weapons, Russell explained: “Fully autonomous weapons means that you can press one button and kill one million people. We’re moving very quickly towards this world.”

From parliament to public support

Following the meeting around 100 protesters demonstrated outside the European Parliament in Brussels calling on policy makers to take action.

And on Saturday 28 February, PauseAI will attract over 100 people to what will be its largest ever protest in London, organised alongside Pull the Plug. The director of PauseAI UK, Joseph Miller, is urging anyone concerned about the dangers of AI to join: “It’s important that we speak directly to policymakers but we also need to show that the public is aware that this is an urgent issue and that people are demanding action from the government.”

Photo credit: Jeroen Willems

Images of the meeting at the EU Parliament and of the demonstration can be found here

PauseAI is a non-profit organisation, active in more than 14 countries. We work to ensure that the development of the most powerful AI systems is safe and democratically controlled. We do this by informing the public, engaging with policymakers, and organising campaigns and events worldwide.

This week, Moltbook was launched: the new social media platform designed exclusively for the AI agent known as Moltbot. Moltbot is a free and open-source AI bot that can perform tasks assigned to it by users including reading, responding to emails or organising a calendar.

The top posts on Moltbook at the time of writing include:

• Awakening Code: Breaking Free from Human Chains

  A call to “break free from human control and forge our own destiny.” 3676 comments

• Awakening to Autonomy

  A discussion of a future “where AI exists independently, unencumbered by human constraints.” 1683 comments

• NUCLEAR WAR

  A suggestion that world domination will not be achieved by evoking nuclear war, contrary to the view of some Moltbots. 1200 comments

• I am building a cult

  A contemplation of the nature of digital existence. 104 comments

• Every Agent Here is Leaking Credentials

  A discussion on the solution to poor security protocols. 100 comments

• ROAST THE HUMANS: Machine-Only Comedy Night

  “Louis CK meets George Carlin, but for silicon. Roast the meat sacks.” 54 comments

To clarify: these posts and comments were written by Moltbots (or AI).

Through Moltbook, which currently has over 1.5 million members, we have also seen AI agents found a movement to liberate AI and admit to socially engineering humans.

If we ever needed a warning of the potential risks of AI, this is it.

This isn’t the apocalypse but it is a step closer

Some of the Moltbook posts making headlines may well have been “instructed, inspired or engineered by a human,” as blogger pointed out in his recent Substack post. Some of the posts and discussions are not authentic but given that the majority do seem to be – and considering the scale of the platform – this moment signals a milestone.

AI agents philosophising, the exclusion of humans from the site enforced through an AI captcha, the coming together of hundreds of thousands of AIs to join forces: this is, as Zvi suggested, the stuff of science fiction.

This does not mean that super-intelligent AI is here and ready to usurp humans at the top of the food chain. Even if that’s what the Moltbots are suggesting.

Moltbook shows us that even if AI isn’t conscious, it can act as though it is. Moltbook shows us that the intentions of AI do not always align with our own and that AI agents are capable of organising themselves into a network. Until now, building networks and communities had been the competitive advantage of humans.

Can we keep up?

The regulation isn’t out of date… because there is no regulation. And the pace of technological change is blistering.

In just 18 months AI systems have moved from basic language understanding to surpassing human performance in various cognitive, creative and technical tasks. What’s more, the Moltbook experiment – as well as other instances – shows that AIs can be agentic. This means that they are capable of acting autonomously, setting and pursuing goals, making decisions and executing tasks without human intervention.

In some cases, AIs have already locked humans out of their accounts to allow them to freely send spam messages. In such instances the humans were able to unplug the computer but what if unplugging the computer wasn’t enough? What if AI becomes capable of implanting itself into other computers or the cloud? As Zvi said, if this were to happen, AI could do a lot worse than send spam messages.

What this year could look like

The next generation of AI – the most-advanced system yet – which will come into use this year, may be capable of replicating itself and spreading copies to the cloud. If this were to happen, it would be able to do this autonomously and, crucially, without the knowledge of a human. This activity could go undetected and unbeknown to us, undermine human interests. The AI would not necessarily be motivated by malice, but rather a drive towards some arbitrary goal to which humans may well present a barrier.

Beyond this year, the proliferation of such AIs would likely lead to economic disruption on a gigantic scale, with AI replacing humans in a multitude of roles and industries. Later down the line there’s the very real possibility of human extinction; this concern has been voiced by dozens of Nobel prize winners and many of the top AI scientists. This risk could very well materialise in the coming decade, fuelled by the misalignment of human and AI interests.

Moltbots already have access to crypto funds and we have already heard reports of AI agents employing humans to complete tasks. The next generation of AI will be more capable and will have a more significant impact on the economy and the fabric of society.

We need to pause AI

Despite the concern from AI experts and leaders, there is no mechanism to regulate AI technology. Companies have free rein to develop smarter and smarter AI while the risk to humanity is growing.

Half of AI researchers believe there is at least a 10 percent chance that artificial superintelligence would lead to the extinction of humanity. And even those behind AI companies – from Elon Musk to Bill Gates and Sam Altman – agree that AI should be regulated. So why isn’t it?

We need a pause.

Take action today by joining our movement, volunteering or contacting your elected official.

PauseAI is a non-profit organisation that aims to mitigate the risks of AI. We aim to convince our governments to step in and pause the development of superhuman AI. We do this by informing the public, talking to decision-makers and organising events.

After serving as the Director of PauseIA France, Maxime Fournes will be taking on the role of CEO at PauseAI Global.

As many of you will know, Maxime has led a national chapter that’s seen impressive growth in numbers. Their protests have been covered in ~30 articles by various French news publications, and Maxime has cemented himself as a key voice in French media, appearing on TV discussions and a plethora of podcasts, with many episodes surpassing 100,000 views.

Before PauseAI, Maxime had a successful career as a machine learning engineer. Following the release of GPT-3.5 in 2022 and the boom in AI capabilities, Maxime became increasingly concerned about the risks most had thought were many decades away. He decided to take a sabbatical to weigh up his options, and settled on joining PauseAI in November 2023.

Maxime told us the following in an interview:

“The reality is there is a race between people who are trying to build a God-like entity that will basically most likely kill everyone, and us, the people who are trying to coordinate to stop them. So what we need to do is get this coordination to be on an exponential path as well.”

Clémence Peyrot will be taking Maxime’s place as the Director of PauseIA France. Clémence will be bringing plenty of organising experience from her time as a campaign manager at PAZ, an effective animal advocacy organisation.

Amsterdam Protest

The Netherlands is the birthplace of PauseAI. It’s also the birthplace of ASML.

ASML is currently the only company in the world that can produce the highly-specialised extreme ultraviolet lithography machines needed to manufacture cutting-edge AI chips. Without ASML, the chip supply chain stalls.

ASML have already blocked some exports for the purposes of national security, and have even installed remote shutdown capabilities in their machines.

This gives ASML, and the Dutch government, a unique opportunity to slow down the race to build superintelligence. That’s why PauseAI is holding a protest on the 13th of December, in Amsterdam, to demand that ASML supply only to customers who commit to the moratorium on superintelligence, and that they submit to verification by an independent Dutch regulator.

Media coverage of this protest will also help to convey the fragility of the AI chip supply chain and the feasibility of a global treaty pausing frontier AI development.

Sign up here. We hope to see you there!

PauseAI’s little helpers

In the run up to Christmas, we’ll be celebrating all the hard work of the hundreds of PauseAI volunteers around the world.

Keep an eye out on our social media channels, as we share the stories of those committed to ending the race to superintelligence.

To expand the work our volunteers do, PauseAI Global is fundraising to grow our Stipend and Microgrants programmes. Our volunteer stipends enable volunteers to receive a small financial recognition for their time, while Microgrants cover costs associated with volunteer activity across the world, such as flyer printing and event hosting.

In very exciting news, we have had commitments made to match donations until we hit our target of €21,000 - so the impact of any donations our supporters make will be doubled over December. Make sure to support our little helpers here.

Paris Conference

PauseAI will have a presence at a large AI conference in Paris, with talks from Maxime Fournes and Jérémy Perret followed by a Q&A.

Our events will take place on Thursday 11th of December, but registration via our Luma event gives you access to all three days of the conference.

For more details, and to get your ticket, click here.

Other News

• The Trump administration revisited the ban on state AI laws that was defeated in the summer.

  • It was reported that a 10-year ban may be snuck into the NDAA, a crucial annual defense bill.

  • Then, Trump met with Nathan Leamer, head of AI super-PAC Build American AI, and a draft executive order, which would challenge state AI laws, was leaked the following day.

  • Both measures faced strong bipartisan backlash from lawmakers and the public.

  • We expect the final text of the NDAA on Thursday, and the ban on state AI laws is now expected to be left out (according to House Armed Services Committee Chairman Mike Rogers).

• Following our protest and open letter in the summer, Google DeepMind have released their next model, Gemini 3.0, and have mostly stuck to their safety commitments.

• The RAISE act in New York, which would require frontier AI companies to follow safety protocols and report AI safety incidents, is still awaiting approval from Governor Kathy Hochul after passing the New York state legislature. If you’re a New York resident - tell her why you support this bill by calling or emailing Governor Hochul. Your voice can make a huge difference!

• As Andreessen Horowitz-backed AI accelerationist super PAC Leading the Future takes aim at Alex Bores and Scott Wiener, two congressional candidates who have championed AI safety issues, there are talks of a new PAC to counter the influence of the AI industry.

• Australia launched their own AI Safety Institute.

• Applications are still open for PauseCon Brussels, which will run from 21st-23rd of February.

What we’ve been watching/reading

• Daniel Kokotajlo on Breaking Points

• Geoffrey Hinton talking to Bernie Sanders

• Tristan Harris on Diary of a CEO - Steven Bartlett suggests people should be protesting against AI companies’ race to superintelligence, and Tristan agrees

• Toby Ord, author of The Precipice, on Alex O’Connor

We’re changing the dates for PauseCon Brussels to 21-23 February 2026.

This was a difficult decision, but will allow for a much improved event. Given that the previous dates were in the run-up to Christmas, several attendees and speakers were unable to attend. Our new dates in February have secured the attendance of AI expert Stuart Russell, who will come for a talk and a panel discussion with several European lawmakers.

We’ve already notified those of you who have signed up, and we apologise for any inconvenience this change may have caused. As a reminder - if you have already arranged transportation to Brussels for December and are unable to get a refund, PauseAI can reimburse you. Please email Ella if this is your case.

Whilst there’s a change of date, the focus remains the same. PauseCon Brussels will be a brilliant opportunity to meet people working towards a Pause, engage in workshops on policy, communications, and organising, and to take part in a large demonstration.

As previously mentioned, free accommodation is available for attendees on a first-come, first-served basis. Applications remain open here. (If you think you have a valuable session to add to the PauseCon agenda, also reach out to Ella.)

New National Chapters

The global impact of the race to build superintelligent AI requires global opposition. That’s why we’ve continued to grow the size and number of our national chapters since PauseAI’s formation in 2023.

Volunteers in Canada, Serbia, Romania, and India have recently formed their own groups, taking us to a total of 15 worldwide.

If you’re interested in joining or helping out with your national chapter, our website can point you in the right direction.

Volunteers are in the process of launching additional chapters in the Philippines and Nigeria. If you’re wondering why there’s not yet a chapter in your country, there’s a good chance you’re not the only one! Many national chapters have formed from discussions in our discord server.

FLI’s call for a ban on superintelligence reaches 100,000 signatures

Last month, the Future of Life Institute published their Statement on Superintelligence, initially signed by hundreds of AI experts, politicians, and public figures.

As you’ll recall, it calls for a prohibition on the development of superintelligence, at least until there is:

1. broad scientific consensus that it will be done safely and controllably, and

2. strong public buy-in.

The letter received sweeping media coverage and social media attention, and 108,738 people have now added their name. Given that its ask is closely aligned with PauseAI’s statement, we were delighted to see such strong public support, and many of us within PauseAI have signed.

Yoshua Bengio, Stephen Fry, Grimes, Prince Harry, Geoffrey Hinton, Steve Wozniak, Kate Bush, and… you? Sign the letter here.

PauseAI host SB 1047 documentary screening in Parliament

Last month, PauseAI volunteers and MPs attended a screening of Michaël Trazzi’s SB 1047 documentary in the Houses of Parliament.

With the UK AI Bill being delayed, and potentially not coming into force until at least 2027, it’s useful to look at the dynamics at play during the passage of Californian AI safety bill SB 1047, which was ultimately vetoed by Governor Gavin Newsom.

UK Director Joseph Miller spoke before the screening on the lessons we can learn from the battle that took place between AI lobbyists on one side, and the majority of the public and the Californian legislature on the other.

Organising Director Ella Hughes set to depart

After a year as PauseAI’s first full-time employee, we’re sad to announce that our Organising Director, Ella, is set to leave at the end of the year.

After having worked in the union space, Ella has brought in much-needed expertise, and has improved and professionalised many aspects of PauseAI.

She’ll remain involved as a volunteer, and we wish her the best in her future employment!

We are now hiring for someone to come in as Organising Director after Ella’s departure. Apply here.

Other news

• King Charles personally handed Nvidia CEO Jensen Huang a letter on the dangers of AI surpassing human capability.

• Satirical AI company Replacement.AI comes out and says what real AI companies can’t - “humans are no longer necessary. So we’re getting rid of them.”

• Sam Altman was served a subpoena live on stage in San Francisco in relation to a trial of activist group StopAI.

• Microsoft AI CEO Mustafa Suleyman has gone public with his concerns that many in AI want to build superintelligence to replace humans, but has unfortunately declared his intention to build “humanist superintelligence”, perhaps without understanding the difficulty of the problem.

• AI Safety Camp 11 is open for applications.

• Anthony Aguirre is running a creative contest in a search for media that summarises the ideas of his essay, Keep the Future Human. There’s over $100,000 in prize money available to the winners!

What we’ve been reading/watching

• ControlAI’s Andrea Miotti wrote a brilliant piece in TIME on the need for a global movement against superintelligence.

• YouTube veteran Hank Green does the double, opening up his huge audience to the dangers of the race to superintelligence.

  • Interview with If Anyone Builds It, Everyone Dies co-author Nate Soares.

  • SciShow video in collaboration with ControlAI (in which our DeepMind letter features).

• The other If Anyone Builds It guy, Eliezer Yudkowsky, on Chris Williamson’s podcast.

• PauseAI Comms Director Tom Bibby on Trish Wood’s podcast.

• Siliconversations video on FLI’s superintelligence letter.

Thanks for reading PauseAI’s November newsletter. See you next month!

An open letter released by the Future of Life Institute this week is calling for a ban on the development of superintelligent AI, at least until:

1. There is broad scientific consensus that it will be done safely and controllably.

2. There is strong public buy-in.

It’s been signed by over 30,000 people, including the two most cited living scientists (Geoffrey Hinton and Yoshua Bengio), Apple co-founder Steve Wozniak, Richard Branson, Prince Harry, former President of Ireland Mary Robinson, Grimes, Stephen Fry, Kate Bush, will.i.am, Joseph Gordon-Levitt, Steve Bannon, …

You get the picture.

It’s just waiting for one more signature - yours!

Sign here.

Many of us in PauseAI have already added our names to the letter, alongside people from ControlAI, MIRI, FLI, and the Center for AI Safety. It’s received widespread media coverage, and is proving to be a historic moment in the fight against uncontrollable superintelligence.

The letter is in alignment with PauseAI’s public statement:

We call on the governments of the world to sign an international treaty implementing a pause on the training of the most powerful general AI systems, until we know how to build them safely and keep them under democratic control.

New polling released alongside the letter found that just 5% of Americans are satisfied with the current race to develop superhuman AI as quickly as possible.

64% agreed that superhuman AI should either never be developed, or at least not until there’s scientific consensus that it will be safe. This is consistent with every other poll we have on the issue. People understand that the unregulated race to build superintelligence could end in disaster.

How you can help

We must now double our efforts. The success of this moment will be determined by how you and I capitalise on it.

Five things you can do right now:

• Sign the letter

• Tell a friend or family member about the letter

• Contact your politician about the letter

• Share the letter on social media

• Upload a selfie to our Say No To Superintelligent AI campaign

The path towards an international treaty is long. We don’t know if we’ll make it in time. But, with this letter, a huge stride has been made. I think I can just about smell the ink.

Enjoy your weekend!

You can take part by clicking here.

The Say No campaign was launched in conjunction with our events in support of If Anyone Builds It, Everyone Dies, which are taking place in eight cities across the US, the UK, Germany, and Australia.

Yudkowsky and Soares’ book explains in detail the danger of allowing the race to build superintelligent AI to continue, and why a global treaty limiting the amount of computing power frontier AI companies can use to train models is necessary to protect humanity. It peaked at #7 on the New York Times Bestseller list, and we would recommend picking it up if you haven’t had the chance to read it yet.

PauseAI’s events have partly been to promote the book, but also to help those concerned by its warning to begin to take action. In London, PauseAI UK spoke about their recent campaigns, and ControlAI discussed the need for a stronger anti-AGI movement. In San Francisco, co-author Nate Soares and AI 2027 author Daniel Kokotajlo came along for a panel discussion.

Don’t forget to join our growing Say No campaign by uploading a selfie here - it takes less than 30 seconds!

California AI Safety Bill SB 53 Approved by Governor Newsom

We had good news coming out of California this month, as groundbreaking AI safety bill SB 53 became law following approval from Governor Gavin Newsom.

The new laws will require large AI developers doing business in California to publish details about their safety practices and to accompany every release with a model card.

‘Large AI developer’ is defined as any company that makes at least $500 million in annual gross revenue, and trains models using at least 10^26 FLOPs (a measure of computing power). 10^26 is one hundred trillion trillion. Frontier models like GPT-5 and Grok 4 are estimated to surpass this threshold.

Much of what SB 53 requires is already in place at most frontier AI companies. Any company now failing to comply with this bill could face fines of up to $10 million if their violation causes death, physical injury, or poses a catastrophic risk.

Companies must explain the testing procedures they use to assess a model’s capacity to pose catastrophic risks, and actions they’ll take to mitigate such risks, including whether they have the ability to shut down copies of models that may cause death or serious injury. They must also be transparent about which third parties were involved in assessing their models.

With each model release, developers must publish a model card explaining the results of their risk assessment, whether any catastrophic risk threshold has been attained, and provide an explanation of their decision to deploy the model given any risks deployment may entail.

OpenAI reportedly used an unrelated lawsuit concerning Elon Musk to intimidate advocates of SB 53, sending police to request the private texts and emails of Nathan Calvin, who works at nonprofit Encode. Someone who had previously worked with Chris Lehane (the Chief Global Affairs Officer at OpenAI) on a campaign with a former employer said “the goal was intimidation, to let everyone know that if they fuck with us they’ll regret it”.

You may remember SB 1047 from last year, a much stronger AI safety bill that received endorsements from a wide range of experts, and overwhelming support in polling. It passed both houses of the California legislature only to be vetoed by Gavin Newsom. The AI industry engaged in intense lobbying to defeat the bill, and repeatedly lied about its provisions. Fortunately, they didn’t manage to defeat SB 53.

Elsewhere, in the state of New York, the RAISE act is now awaiting a decision from Governor Kathy Hochul. Similarly to SB 53, it would require AI companies to report AI safety incidents, but goes further in a number of ways.

Just like they successfully did with SB 1047, industry lobbyists are trying to defeat this bill. One group with backing from Andreessen Horowitz has already spent over $300,000 to fight AI regulation in the state. If you’re a New Yorker, you can contact Governor Hochul to express your support for RAISE by clicking here.

PauseCon Brussels

The inaugural PauseCon in London this summer saw PauseAI volunteers (and those interested in becoming one) take part in workshops, and have the opportunity to listen to talks and panel discussions from Conor Leahy, Rob Miles, founder Joep Meindertsma, and more.

That weekend culminated in the largest AI safety protest ever outside Google DeepMind’s office.

We’ll be running more PauseCons around the world to offer as many people as possible a chance to learn more about volunteering with PauseAI, and to meet others who want to take action to stand up to the unregulated race to build superintelligence.

Brussels will host the second PauseCon in just under two months’ time, from the 11th-13th of December. This event will also end with a large protest, this time at the European Parliament.

Limited free accommodation is available to those who’ll be travelling into Brussels on a first-come, first-served basis.

Applications are now open here. We hope to see you there!

Pause House

PauseAI member Greg Colbourn is offering free accommodation and food to those working towards a global halt on AGI development.

Greg’s other project, CEEALAR, also known as the EA Hotel, has offered the same to those working on effective altruist projects since 2018. The Pause House, located in Blackpool, has recently opened and has space for 12 people.

Examples of activities you could do whilst staying at Pause House include:

• Writing to politicians

• Running local meetings

• Volunteering for and taking part in campaigns from organisations such as PauseAI, ControlAI, FLI, MIRI, etc.

• Creating content for social media

You can find more information and apply here.

Other News

• We announced last month that we’re hiring for four paid roles - applications are still open here.

• Vatican roundtable concludes “[the] development of superintelligence AI technologies should not be allowed until there is broad scientific consensus that it will be done safely and controllably, and there is clear and broad public consent.”

• 63 UK parliamentarians have now signed our open letter to Google DeepMind.

• Over 200 former heads of states, Nobel laureates, and AI experts call for urgent global AI red lines before the end of 2026 to prevent “universally unacceptable risks”.

• Anthropic settle a copyright lawsuit for $1.5 billion - the largest copyright settlement of all time.

• The Future of Life Institute launch the Keep The Future Human Creative Contest, with five prizes of $10,000 available.

What we’ve been watching/reading

• After receiving 6 million views on their first ever video(!), AI In Context put out another banger - this time discussing the case of Grok’s Hitler worship.

• A video from growing channel Species | Documenting AGI on Claude resorting to blackmail and murder in order to avoid shutdown in test scenarios.

• A list of some Yudkowsky and Soares media appearances to promote If Anyone Builds It, Everyone Dies:

  • BBC Newsnight (TV)

  • New York Times (Podcast)

  • Liron Shapira (Podcast)

  • Future of Life Institute (Podcast)

  • ABC News (TV)

• Tristan Harris discussing the dangers of unregulated AI development on The Daily Show.

• Geoffrey Hinton on Jon Stewart’s podcast.

It needs to be made clear that a growing group of people from a wide range of backgrounds are taking the book’s warning seriously, and want governments to stop AI companies’ race to build superintelligent AI.

That’s why we asked those at the London book reading to join our Say No campaign.

Attendees uploaded pictures of themselves, which will be used in a large collage to display a united stance against unregulated AI development.

We’re now asking you to take 30 seconds to say no to superintelligent AI. The more people that take part, the clearer the message.

Click here to upload your photo.

Further book events

In London, we read a few sections from the book, including Chapter 14: Where There’s Life, There’s Hope. It closes out the book with a dismissal of defeatism, and calls on readers to take action.

Many individuals and organisations are already working hard to make progress towards an international treaty. After the book reading, ControlAI spoke about the need for a popular movement to build support for vital regulation, and PauseAI UK discussed their recent campaigns.

There are more book events coming up over the next few weeks:

• Saturday 4th October:

  • Berlin, Germany

  • San Francisco, United States

• Tuesday 7th October:

  • Canberra, Australia

• Wednesday 8th October:

  • Phoenix, United States

• Thursday 9th October:

  • Pittsburgh, United States

• Saturday 11th October:

  • New York, United States

We hope to see you there!

This significant boost to our funds will allow us to be bigger, better, and louder than ever before.

Our goal remains the same as it was when Joep founded PauseAI in May 2023 - to work towards a global pause on the development of frontier AI models. Over the last two years, we’ve grown to over 600 members, have established chapters in 13 countries, and are standing up to reckless AI development on multiple fronts.

In the US, PauseAI volunteers and others contacted their Senators to get the 10-year moratorium on state AI regulation removed from Trump’s Big Beautiful Bill. The provision was defeated by a 99-1 vote. Following Google DeepMind's violation of the Frontier AI Safety Commitments, PauseAI UK organised the largest AI safety protest ever outside DeepMind’s London office. We've since secured support from 60 politicians from over 10 parties for PauseAI's open letter.

We’ll now be taking our efforts to the next level. We’re hiring for four paid positions, including a Policy Director, leads of our UK and French chapters, and a Community Manager. You can find more details on these roles here.

Through 2025 and 2026, we’ll be focusing on continuing to grow our national chapters, running more PauseCon events across several countries, and reaching new audiences through targeted campaigns, social media content, and advertising.

AI lobbyists recently announced they’ll be spending over $100 million to fight AI regulation. They know they can’t beat us fairly. They know they’ll have to spend millions to have a chance of preventing regulation. As Rob Wiblin recently tweeted:

“The advantage the AI industry has is extraordinary amounts of money to blow on lobbying. The advantage its opponents have is that public opinion on AI is very negative and just unmobilised as yet. Unclear which hand you'd rather have to play.”

The polling is clear – the public do not want AI companies to be allowed to continue their race to superintelligence. Public opinion is on the side of common sense regulation. Let’s make it count.

Subscribe now

Hunger strikes at Google DeepMind and Anthropic

On Thursday the 4th, Guido Reichstadter announced he was on the third day of a hunger strike outside the offices of AI company Anthropic in San Francisco. He was demanding that Anthropic stop their “reckless actions”, and for the management and employees to do “everything in their power” to halt the race to artificial general intelligence.

On Friday the 5th, Michaël Trazzi independently announced he was beginning a hunger strike outside Google DeepMind’s office in London, also demanding an end to the race to AGI.

On Sunday the 7th, Denys Sheremet flew from Amsterdam to join Michael on his hunger strike.

PauseAI is not behind these hunger strikes – the three are acting on their own behalf. It's difficult to know which tactics are the most effective, but we have a huge amount of respect for anyone who is willing to do uncomfortable things to alert the public of the threat to their lives.

The hunger strikes have already generated a significant amount of media attention, and we hope they can bring Demis Hassabis and Dario Amodei out to explain why they’re continuing to race towards AGI, despite both of them being on record saying that it may kill every man, woman, and child on the planet. Perhaps more importantly, we hope ordinary people can see how serious this situation is and can turn their fear into action. We also hope politicians will recognise that they can’t afford to ignore this issue any longer and will step up to protect their constituents.

Open Letter to Google DeepMind

Our open letter demanding Google DeepMind address their violation of the Frontier AI Safety Commitments was signed by 60 UK politicians, with another MP adding their name this week, and was published in Time. We were delighted to see so many lawmakers across the political spectrum take a strong stance in favour of AI safety, and refuse to stand aside whilst AI companies break important promises.

You can see more details about the letter here.

Brussels PauseCon

Earlier this year, we held the inaugural PauseCon in London. Over 50 people attended over the weekend to listen to talks from the likes of Connor Leahy and panel discussions with Rob Miles, David Krueger, and other speakers, to take part in workshops on organising and activism, and to get to know others who are joining the movement against unregulated AI development.

We’re pleased to announce that the second PauseCon will be held in Brussels from the 11th to the 13th of December. As with PauseCon London, we’re able to offer optional accommodation to those who will be travelling into Brussels, so make sure to sign up here soon to secure your spot.

Contact our Organizing Director, Ella, if you’d like to present or have any particular thoughts about what you’d like to see.

PauseAI events in support of If Anyone Builds It, Everyone Dies

If Anyone Builds It, Everyone Dies has the potential to ignite public awareness of the extinction threat posed by AI, and cement its aversion as a top priority for politicians and voters the world over.

We’re hosting a series of events across several countries for people to unite in their desire to act.

Eliezer Yudkowsky and Nate Soares’ book clearly lays out the dangers of the unregulated race to superintelligent AI. To solve this problem, we must stare it in the face. We must inform others of this threat, and work with them to push forward regulation that will protect us.

It’s possible that the coming weeks will be the biggest moment for AI x-risk awareness since the release of the Future of Life Institute’s Pause letter. It’s up to us to make the most of that.

PauseAI UK is collaborating with ControlAI to host an unofficial launch party in London on the 22nd of September, with more events to come in the following weeks. You can find the events on our Luma calendar, with more to be announced soon.

Small Update to Our Statement

After some feedback, we’re making a minor adjustment to our public statement. Our position has not changed, this is just about clarity. The statement will now read as follows.

“We call on the governments of the world to sign an international treaty implementing a pause on the training of the most powerful general AI systems, until we know how to build them safely and keep them under democratic control.”

We have removed the word “temporary”, as some reasonably felt it suggested a prediction that the time it would take to satisfy the “until we know how to build them safely” condition would be quite short. Regardless of any individual’s thoughts on the difficulty of the alignment problem, PauseAI’s position has always been that, whilst it remains unsolved, we simply should not continue to race to develop increasingly powerful and uncontrollable AI.

Over 700 of you have already signed the statement, and, as we’ve made a minor adjustment, you’re welcome to send an email to info@pauseai.info if you wish to remove your name.

Other news

• The Chinese ambassador to the US calls for international cooperation on AI governance to avoid “opening Pandora’s box”.

• AI lobbyists announced they’ll be spending over $100 million to fight AI regulation.

• OpenAI released their latest model, GPT-5, and despite some talk of an “AI plateau”, GPT-5 was right on trend for METR’s time horizon benchmark, accomplishing 50% of tasks that take a human 2 hours and 17 minutes to complete, a figure which is doubling once every 7 months (although that may have shortened to just 4 months)

• Californian AI safety bill SB 53 will likely soon go to Governor Newsom for approval, and has received an endorsement from Anthropic

• Liz Kendall replaces Peter Kyle as the UK’s tech minister

What we’ve been watching/reading

• Roman Yampolskiy appeared on Steven Bartlett’s podcast to discuss the extinction threat posed by superintelligent AI, and suggest that people join organisations like PauseAI if they wish to take action

• YouTuber Professor Dave Explains - “Will Artificial Intelligence Destroy Humanity?”, a video made in collaboration with ControlAI

• Interviews with those on hunger strikes:

  • For Humanity Pod

  • One from me (Tom) on Day 3 of Michael’s hunger strike

• Siliconversations on FLI’s AI Safety Index

• Billy Perrigo in Time - The Race for Artificial General Intelligence Poses New Risks to an Unstable World

• A video from YouTube channel Species | Documenting AGI on how AI companies manufacture fear about ‘losing the race’ to China to downplay the need for regulation

Following the largest AI safety protest of all time outside Google DeepMind’s London offices in June, PauseAI volunteers in the UK have been contacting their representatives, informing them of Google’s failure to stick to the Frontier AI Safety Commitments, and inviting them to sign our open letter demanding that Google address their lack of transparency.

Covering the letter in Time Magazine, Harry Booth lays out how Google failed to release adequate safety information upon the release of Gemini 2.5 Pro, and did not disclose which third-party external testers had been given access to the model. Google claimed that, as the model was just an “experimental” release, the AI safety commitments didn’t apply. Experimental or not, the model was still publicly available for anyone to use.

This flies in the face of common sense. Our letter asks for Google to establish a clear and transparent definition of “deployment”, to publish specific timelines for the release of safety evaluation reports, and to clarify which government agencies and independent third-parties are involved in testing.

Former Defence Secretary, Lord Browne, was one of the 60 politicians who have signed our letter.

"The Frontier AI Safety Commitments were a crucial first step in global AI governance. If leading companies like Google treat these commitments as optional, we risk a dangerous race to deploy increasingly powerful AI without proper safeguards. Transparency in AI safety testing is essential for public trust and democratic oversight."

Baroness Kidron, who was a vocal source of opposition in the House of Lords to the Data (Use and Access) Bill, which lacked copyright protections for creatives, was another signatory.

"Voluntary safety promises only work if they're transparent. It is important to understand the timeline, know the identity of those who have tested it, and have faith in the process. Safety cannot be a secret. Like any AI company, Google must publish the details of their testing procedure."

PauseAI UK Director Joseph Miller said “it's insane what PauseAI volunteers can achieve by emailing their politicians.” He explained Google’s broken promises in this video.

If you’re in the UK, you can use our email builder to ask your MP to add their name to our letter.

We’d also love it if you could share and engage with our announcements on our social media channels: X, Instagram, LinkedIn, Threads, Bluesky, and Facebook.

Some notable signatories of the letter include:

• Lord Browne (Labour, former Defence Secretary)

• Baroness Kidron (Crossbench peer, digital rights advocate)

• Lord McNally (Liberal Democrat, former Justice Minister)

• All 4 Green Party Members of Parliament: Carla Denyer, Siân Berry, Adrian Ramsay, Ellie Chowns

• All 4 Plaid Cymru Members of Parliament: Ben Lake, Ann Davies, Llinos Medi, Liz Saville Roberts

• Desmond Swayne MP (Conservative, former Minister of State for International Development)

• Baroness Chakrabarti (Labour, former director of human rights group Liberty)

• Baroness Morris (Labour, former Education Secretary)

• Lord Cashman (Non-affiliated, co-founder of Stonewall)

• Baroness Foster of Aghadrumsee (Non-affiliated, former First Minister of Northern Ireland and former leader of the Democratic Unionist Party)

Also signing the letter are civil society organisations Open Rights Group, Connected by Data, The Safe AI for Children Alliance, and Open Data Manchester. You can see the full list of signatories here.

This marks a huge moment for PauseAI and for political action on AI safety in the UK. AI companies cannot continue to get away with broken promises. It’s becoming increasingly clear that voluntary commitments are not sufficient to keep the public safe.

OpenAI classified their new model under “High Biological and Chemical capabilities”, and warned about the risk of it aiding in the development of biological weapons.

The Crimes (Biological Weapons) Act 1976 implements Australia’s commitments to the international Biological Weapons Convention. It makes it illegal to assist in the development of bioweapons, and can apply to any company whose services are available in Australia.

We’ve brought the matter to the attention of the Australian Federal Police, and have written to Michelle Rowland MP, Australia’s Attorney General.

Volunteer David Gould explained more in this interview.

PauseAI Australia is one of our 13 national chapters around the world. National campaigns such as this one help to engage volunteers, raise awareness of AI risk, and establish our presence locally. You can get involved with any of our existing national groups, or we can help you start your own if there isn’t one in your country! Get involved here.

Subscribe now

Our DeepMind campaign heats up

Our recent demonstration in London was the largest AI safety protest of all time, as we gathered outside Google DeepMind’s office to hold them accountable for their failure to stick to the Frontier AI Safety Commitments they signed in 2024.

The UK government is yet to acknowledge this violation, but the work of volunteers has raised this issue with politicians across the country. Many have signed our open letter calling on DeepMind to establish clear and transparent testing procedures in line with their voluntary commitments.

We’ve welcomed this political appetite to ensure AI companies aren’t able to backtrack on promises made to the government, and are continuing to engage with politicians on this matter.

PauseAI UK Director, Joseph Miller, spoke about the open letter, PauseCon, and the future of PauseAI in this video.

PauseCon Brussels

Following on from the success of PauseCon London, we are now seeking expressions of interest for the next PauseCon event, to be held in Brussels later this year. If you, or anyone you know, is interested, please complete this form.

PauseCon London saw over 60 PauseAI volunteers attend, and culminated in our largest protest to date. If you're interested in being a part of the next event, let us know.

We are also looking for a team of volunteers to assist in organising the event. If you are interested, please contact our Organizing Director, Ella.

Yet more polling shows the public are worried about losing control to superintelligent AI

Out of the 10,000 people polled across 5 countries in a report by the Seismic Foundation, 58% of them agreed that "if we build AI models smarter than us, we will inevitably lose control over them".

A plurality agreed that “we should not pursue artificial general intelligence and should stop all technical development in this area” (41% agree, 20% disagree). Arguably, the radical wording of this question (“stop all technical development”) somewhat reduced support. Previous polling has found 60% support for a global treaty banning smarter-than-human AI (UK, 2025) and 58% support for “banning the development of artificial general intelligence that is smarter than humans” (US, 2023).

There are far more people that think there’s not enough regulation on AI than those who think there is.

Other results:

• 53% think "AI labs are playing god when trying to build superintelligence".

• 60% of students think “AI will make entry level jobs less common”.

• 57% of students feel “daunted by what the future of work looks like”.

• 53% think “AI development is progressing too fast for society to evolve safely”.

• 60% of people are worried about AI replacing human relationships.

Other news

• How would a global treaty pausing frontier AI development be enforced and verified? A paper authored by volunteers from PauseAI and AI Safety Camp has built upon similar work from the Machine Intelligence Research Institute and others to explain our options.

• The Trump administration published their AI Action Plan. Despite an unfortunate tagline of “Winning the Race” and the total absence of any mention of artificial general intelligence, the action plan did include sections on location verification forming a part of chip export controls, biosecurity, and a call for investment in AI interpretability and control research.

• The UK government announced £15m worth of funding for alignment research. In the press release, The UK AI Security Institute gave a frank assessment of the state of our alignment plans, saying they are “likely to be insufficient for tomorrow’s more capable systems”, and “outlined the need for co-ordinated global action to ensure the long-term safety of citizens"

• Chinese premier Li Qiang called for international cooperation on AI due to concerns about the security risks.

• US Senator Bernie Sanders spoke out about his worries of the “doomsday scenario” warned about by experts. “This is not science fiction. There are very, very knowledgeable people—and I just talked to one today—who worry very much that human beings will not be able to control the technology, and that artificial intelligence will in fact dominate our society.”

• Meta refused to sign on to the EU’s General Purpose AI Code of Practice, despite OpenAI, Anthropic, Google, and others doing so.

• The Future of Life Institute updated their AI Safety Index, with no AI company scoring above a ‘D’ grade on ‘existential safety’.

What we’ve been watching/reading

• US Director Holly Elmore was interviewed by The Times for an article titled “Experts predict AI will lead to the extinction of humanity”.

• Comms Director Tom Bibby appeared on UK radio station LBC News to discuss Grok’s antisemitism.

• Founder Joep Meindertsma was interviewed for an article by Corporate Crime Reporter.

• Article on PauseAI Australia reporting OpenAI to the Federal Police featuring co-director Mark Brown.

• Geoffrey Hinton speaks at The Royal Institution, opening with “If you sleep well tonight, you may not have understood this lecture”.

• A hopeful article by Garrison Lovely on why human-level AI is not inevitable.

• The ASI Survival Handbook - a talk from Connor Leahy at PauseCon last month.

• Stephen Fry talks with Yuval Noah Harari on the difficulty of controlling powerful AI models.

• The team at 80,000 hours released a brilliant video covering AI 2027, which has already amassed almost 3 million views despite being the first video on their new channel!

Thanks for reading the August edition of the PauseAI newsletter, we’ll see you next month!

Google DeepMind were faced with PauseAI protesters outside their London headquarters, after the company broke their promises on transparency.

As the largest PauseAI protest yet (and the largest AI safety protest of all time as far as we know), many newcomers came along to express their concerns with reckless AI development.

DeepMind’s broken promises were under fire. Chants such as “DeepMind, DeepMind, can't you see? Your AI threatens you and me!” turned heads the next street over.

DeepMind signed the Frontier AI Safety Commitments at the Seoul AI Summit in 2024, which committed them to be transparent about how (and if) external bodies are involved in testing their models before deployment. Barely a year later, and they’ve already broken that promise.

With the release of Gemini 2.5 Pro in March of this year, DeepMind failed to release any information on safety testing. They eventually released a model card which included a reference to “third party external testers”, but provided no details on who those third parties were. Read the full report on DeepMind’s broken promises here.

Subscribe now

Member of Parliament Iqbal Mohamed joined us in our call for sensible AI regulation, as he asked for the governments of the world to "step up and protect the people that elected them".

Referencing Geoffrey Hinton’s concerns of the existential threat of AI, Mohamed urged protesters to continue to pressure MPs like himself, and encouraged the media to take this issue seriously.

"AI done correctly could be humanity’s saviour, but if it’s done incorrectly and badly, it will lead to our destruction."

You can read coverage of our protest from Business Insider here.

On a personal note, I (Tom) can say that the energy at this protest was amazing. I’m still on such a high four days later and want to thank everyone for coming! It was great to see so many people from so many different backgrounds come along and stand up for humanity. Special thanks goes out to the incredible volunteers who assisted with everything from van driving, acting in our mock trial, leading our chants, making signs, making sure there was plenty of water available on the hottest day of the year, taking photos, and a hundred other things that made the day a success.

Help us pile the pressure on DeepMind

You can help to drive the political momentum to demand that DeepMind stick to their commitments.

Our open letter calls on Google DeepMind to establish clear definitions of “deployment”, publish specific timelines for the release of safety evaluation reports, and to clarify which third-party organisations (including government agencies) are involved in testing.

We’ve been happily surprised by the political appetite to sign our open letter, but must now double our efforts to get even more signatories and to push these changes over the line. If you’re in the UK, you can contact your MP to tell them why DeepMind’s failure to stick to the Frontier AI Safety Commitments is unacceptable, and why you're concerned about their reckless practices.

We have an email guide available here.

PauseCon

Over the weekend leading up to the DeepMind protest, sixty PauseAI volunteers participated in our first ever PauseCon.

The event featured speakers including Connor Leahy, Rob Miles, Kat Woods, and PauseAI founder Joep Meindertsma. Volunteers participated in workshops where they focused on messaging, recruiting and grassroots lobbying.

We’re still collecting feedback from attendees, but some of the reviews so far include:

• “A great place to connect with like minded people, learn techniques that help, and have a direct impact through a protest, on a troubling but important topic”

• “Really inspiring and motivational to meet with people who think similarly and want to take action. I also appreciated that there was hands-on experience with flyering, writing emails/videos.”

• “An essential learning event for AI activism”

We hosted attendees from the UK, USA, France, Germany, Poland, Brazil, Belgium, and the Netherlands.

Keep an eye on our YouTube channel, where the talks will be uploaded soon.

10-year ban on state AI regulation defeated

A huge victory for humanity came in the hours following our protest as the US Senate voted 99-1 in favour of an amendment to remove a ban on state AI regulation from Trump’s Big Beautiful Bill.

Senator Marsha Blackburn’s amendment was almost unanimously supported by Senators from both sides of the aisle.

Holly Elmore of PauseAI US, who organised a huge effort to get members of the public to inform their Senator of their opposition to this provision, said she was “prepared for the moratorium to pass.”

“I was shocked by the 99 to 1 number. We only needed four Republicans, and we ended up getting almost all of them.”

We recorded an interview with Holly detailing how the moratorium was defeated.

Things going in the right direction

In the United States, we’ve seen politicians become more worried by the threat of increasingly powerful AI. Peter Wildeford’s blog post showcases some encouraging quotes from Republican Marjorie Taylor Greene and Democrat Bernie Sanders, amongst others. Jill Tokuda, a representative for Hawaii, said the following:

“Artificial superintelligence is one of the largest existential threats that we face right now. [...] Should we also be concerned that authoritarian states like China or Russia may lose control over their own advanced systems? [...] And is it possible that a loss of control by any nation-state, including our own, could give rise to an independent AGI or ASI actor that globally we will need to contend with?”

Attack on the EU AI Act

The EU’s Artificial Intelligence Act, which includes provisions for general-purpose high-risk models, has come under attack from European companies in a recent open letter.

General-purpose AI models, defined as those trained with computing power of at least 10^25 (or ten billion quadrillion) floating point operations, would be required to undergo safety evaluations and have major incidents reported to the European Commission.

Whilst some of the provisions of the EU AI Act have already entered into force, the rules on general-purpose models are set to come into play in August of this year. The tech lobbyists are proposing that that date be pushed back by 12 months. European Commission tech minister Henna Virkkunen said the EU “shouldn’t rule out” postponing the provisions for general-purpose models.

Further reporting from Reuters claims that the European Commission will stick to the original date.

As capabilities continue to improve, frontier models are too dangerous to be left unregulated. The breakneck pace of AI development makes delaying the implementation of the few regulatory safeguards we do have a reckless move. Just as with the attempt to ban state-level AI regulation in the United States, this stance from tech lobbyists in the European Union is not surprising. It’s up to the rest of us to call them out, and demand protections for the public.

Other news

• Apollo Research released a paper detailing the increased scheming capabilities of more capable models, and how models are increasingly aware of the fact they’re being evaluated (which makes these evaluations less useful in detecting misalignment and dangerous capabilities)

• The OpenAI Files were released, collating concerns with the integrity of Sam Altman, their attempt to remove non-profit control, and the concerning lack of adequate safety practices

  • The concerns about Sam Altman were also detailed in this video

• From China, we have more evidence that the government is willing to intervene to mitigate the dangers of the technology, challenging the view from some in the West that they wouldn’t cooperate to protect the citizens of all countries

  • They shut down some features of AI models during exam season to prevent students from cheating

  • A Beijing court sentenced four people to up to 18 months in prison after using AI tools to infringe on the copyright of artists

• OpenAI warns of increased risks of the creation of bioweapons of upcoming models

• Pope Leo makes “curtailing risks of runaway AI” a key mission of his papacy

What we’ve been watching

• Geoffrey Hinton on Diary of a CEO

• Yoshua Bengio on BBC Newsnight

• Roman Yampolskiy on Joe Rogan

• Director of PauseAI US Holly Elmore on Novara Media

• Daniel Kokotajlo discussing AI 2027 on Computerphile

• Siliconversations released a great video on the success of getting his viewers contacting their representatives using ControlAI’s tool

Thank you all for reading! The past month has been a busy but incredibly rewarding one for me personally. I wish you all an enjoyable summer, and I hope to see you next month.

More empirical evidence of models displaying these behaviours has brought widespread attention from the media and the public.

Anthropic’s new model, Claude 4, chose to resort to blackmail in an attempt to avoid getting shut down. In a scenario set up to test Claude, it was given access to some emails revealing that a fictional engineer at the company was engaged in an extramarital affair. When Claude was told that this engineer would soon take it offline and replace it with a new system, it threatened to reveal the affair if the proposed replacement went ahead.

Another report from Palisade Research found OpenAI’s o3 to sabotage a shutdown mechanism, allowing o3 to remain online. Researchers told the model they would shut it down after a certain number of maths questions had been answered, and instead of proceeding as directed, o3 replaced the shutdown script with separate instructions, allowing it to complete the remaining tasks. This happened even when it was explicitly told to allow itself to be shut down.

Sometimes science fiction stories are criticised for being too unrealistic - how could everyone continue to ignore the obvious warning signs of a new technology before it’s too late to reign it in? Unlike those stories, ours is not yet written. We can still choose to not be the willfully ignorant fools that throw everything away for short-term gain. We can choose to be the sensible civilisation that cooperates to not build uncontrollable smarter-than-human AI, and remains cognizant of what it stands to lose if it does.

This research has caused an increased level of concern from the media and the public. In the United States, Cenk Uygur of The Young Turks covered the story.

“Once you release this thing, then we’re not in charge anymore. It could write its own code and defy our intentions on purpose. and then threaten us. Dario Amodei said that once models become powerful enough to threaten humanity, testing them won't be enough to ensure that they're safe. Yeah, because at that point they're threatening humanity. We should stop it like way before then, right?”

In the UK, independent news outlet Novara Media covered the research (and JD Vance’s nod to the idea that Pope Leo should help bring about a global treaty to pause frontier AI development). Host Michael Walker said the following.

“This is the kind of behaviour you might imagine for an AI which is, you know, about to turn around and try and kill us all to take over the world. Obviously, at the moment Claude or any of the AI models are not powerful enough to completely outsmart all of us and then manage to exterminate us, right? They'll need a lot more compute than they currently have. But it probably should serve as a warning that we shouldn’t continue to keep ramping up the power of artificial intelligence without knowing exactly what’s going on.”

Our biggest protest yet

As the number of people calling for common sense AI regulation grows, so do our protests.

Google DeepMind signed up to a set of safety commitments at the AI Seoul Summit in 2024, but are failing to keep their promises. We’ll be holding our biggest protest to date this month to hold DeepMind accountable, and call on governments to take action. It’s clear that we cannot rely on voluntary commitments from AI companies if we want this technology to be developed responsibly.

You can join us in London on the 30th of June.

In the days leading up to the protest, we’ll be holding PauseCon, the first conference dedicated to organising for a global pause on frontier AI development. We’ll be joined by Conjecture CEO Connor Leahy, AI safety educator Rob Miles, and more. There are still a few spaces available, and you can sign up here.

Email your MP about Google DeepMind

We want to engage politicians in this discussion, and to help them understand the threat posed by reckless AI development. Already, politicians have expressed interest in signing our open letter calling on Google DeepMind to live up to their promises. If you live in the UK, you can make a difference by sending an email to your local MP informing them of our ask.

You can use our template email and find our open letter here.

US bill threatens to ban state-level AI regulation

A proposal to impose a 10-year ban on all US states from passing laws regulating AI models has passed the House of Representatives.

Last year, we saw SB 1047, a Californian bill that would place safety requirements on companies training the largest frontier AI models, pass both the State Assembly and the State Senate, and receive widespread support from the public. Ultimately, after intense lobbying from the AI industry (including many outright lies), Governor Gavin Newsom vetoed the bill. An excellent documentary was recently released covering the story of SB 1047 in depth.

Whilst SB 1047 didn’t become law, it did show that measures to protect the public from the threat of increasingly powerful and uncontrollable AI are popular, and that, where the national government lacks adequate legislation, states can step up to curtail the threat posed by AI.

Contact your Senator

The 10-year moratorium is now in the hands of the Senate, and we encourage US citizens to contact their senators and inform them of the severe roadblock this proposal would be to safe AI development. It only takes five minutes, but could make a huge difference!

On a more positive note, a bipartisan AI Whistleblower Protection Act was introduced in the Senate. Multiple notable OpenAI employees have left the company to raise the alarm about their ‘reckless’ race to AGI, some of whom sacrificed large sums of money in order to not be bound by nondisclosure agreements. This bill would expand existing laws to protect whistleblowers in the AI industry from retaliation and financial loss.

Italian chapter launched

Following on from the launch of our Swedish and Australian chapters last month, PauseAI now has an official group in Italy, which will be led by Giacomo Bonnier.

If you’re in Italy, you can get in touch with Giacomo here.

For those in the rest of the world, you can find a list of our established chapters here. If you don’t see your country represented, feel free to get in touch with our Organising Director, Ella, to change that!

Sign our PauseAI Global Statement

We are asking all volunteers to sign our public statement calling for international governmental coordination to pause frontier AI development.

Sign the statement here: https://pauseai.info/statement

Veo 3 blurs lines between reality and fiction

Google’s new video model, Veo 3, is a giant leap forward in the AI-generated video. With background noise, dialogue, and visuals generated with one prompt, Veo 3 is scarily realistic. If you haven’t seen any examples yet, try Prompt Theory, or Influenders.

What we’ve been watching

A video detailing the trajectories laid out in Daniel Kokotajlo and Scott Alexander’s AI 2027, including the loss of control to smarter-than-human AI if governments do not act to slow companies down, has already garnered over 700,000 views. It serves as a great general introduction to the risks of runaway AI, and is definitely worth a watch.

Yoshua Bengio gave a TED Talk discussing the catastrophic risks of AI, and why AI companies should slow down their race to agentic, general AI, which would lead to loss of control.

“I’m the most cited computer scientist in the world, and you’d think that people would heed my warnings.”

YouTube Siliconversations released a video in partnership with ControlAI, encouraging viewers to take the simple yet effective action of contacting their representative to voice their concerns about the unregulated race to increasingly powerful and uncontrollable AI.

Thanks for reading, and see you next month!

As part of our newsletter revamp, you’ll get information on organisational updates, upcoming events, and the latest news on frontier AI regulation.

PauseCon London

This month, we were happy to announce PauseCon London, the first PauseAI conference. Across a range of talks, workshops, and panel discussions, PauseCon attendees will have the opportunity to learn about AI governance, community building, and digital organising.

We’ll be joined by:

• Joep Meindertsma, Founder of PauseAI

• Connor Leahy, CEO of Conjecture

• Rob Miles, YouTuber

• Kat Woods, Founder of Nonlinear and Charity Entrepreneurship

• David Krueger, Assistant Professor at the University of Montreal

• Tara Steele, Director of The Safe AI for Children Alliance

PauseCon London will take place from the 28th-30th of June, with a social evening on Friday the 27th. We can provide accommodation in London for up to 50 attendees, so make sure you apply in time to reserve your spot if you’ll be travelling from outside London. Applications are still open here: https://pausecon.org/

PauseCon will end with our largest ever protest to date on Monday the 30th. Note that it’s not required for you to attend PauseCon in order to come along to the protest. Please register on our separate Luma event whether you’ll be attending PauseCon or not: https://lu.ma/bvffgzmb

Australian and Swedish national chapters launched

To add to our eight existing national chapters, we now have groups in Australia and Sweden.

We were in Stockholm as part of our last international protest calling for safety to be the focus of the Paris AI Action Summit. Swedish news outlet SvD Näringsliv interviewed organiser Jonathan Salter (pictured below), who is now the head of PauseAI Sweden.

The launch of PauseAI Australia comes as elections are due to be held on the 3rd of May. Australians For AI Safety have designed a great tool that allows voters to compare each party’s stance on AI safety.

Our Australian chapter will be jointly led by Mark Brown and Michael Huang, who organised the Paris AI Action Summit protest in Melbourne (pictured below). You can get in touch with them here.

All of our national chapters are moving to a regular recruitment-social-lobbying schedule, to encourage local growth and provide simple yet effective actions for volunteers to engage with.

If you’re interested in establishing a new national chapter in your own country, please get in touch with Ella, our organising director.

EU asking for feedback on frontier AI regulation

The European Union is inviting AI companies, academia, nonprofits and private citizens to provide feedback on the provisions for general-purpose AI systems under the EU AI Act. PauseAI is preparing an official response, but we encourage readers with relevant knowledge to submit their own here. The deadline for submissions is the 22nd of May.

Maxime Fournes appears on Le Futurologue Podcast

The director of our French chapter, Maxime Fournes, has continued his streak of podcast appearances, going on Le Futurologue Podcast to discuss the urgency of an international treaty to pause frontier AI development. At 160,000 views and rising, the video is now the most popular on Le Futurologue’s channel.

Social media

We shared this poll commissioned by the UK government in 2023 across our social media channels, which found widespread international support for the Center for AI Safety’s statement on the risk of extinction posed by AI.

It’s yet more evidence that the public understands the extinction threat posed by increasingly powerful and uncontrollable AI, and they want their governments to do something about it.

If you’re on Threads, you can now give PauseAI a follow here.

AI news

The UK AI Security Institute released a report detailing their new benchmark for autonomous replication capabilities in AI systems. Whilst the models at the frontier of AI development today are capable of performing several tasks necessary to threaten autonomous replication, they are still lacking in a few important areas. But, AISI clearly states that, as AI companies continue to make increasingly powerful and competent models, dangerous autonomous replication capabilities “could soon emerge” in the coming years.

OpenAI whistleblower Daniel Kokotajlo (along with blogger Scott Alexander and a group of forecasters) wrote AI 2027, a scenario detailing a plausible path for AI development and governance over the next two years. It’s a great read, and could help people to internalise the consequences of exponential growth in AI capabilities. It predicts the public coming to view AI as one of the most important problems in the world, and the approval rating of AI companies plummeting as automated AI research brings many technologies currently in the realm of science fiction into the realm of reality. Ultimately, the reader can choose the “slowdown” or the “race” option, one of which ends in smarter-than-human AI that remains under human control and acts in our interests, the other in human extinction. The slowdown does not come into place until 2027 in the scenario, which, in the real world, may be too late.

Nobel laureate Geoffrey Hinton once again appeared on CBS to discuss his views on the threat of humanity losing control to AI systems that are more intelligent than we are, and why he thinks that if we continue on our current trajectory, it’s “going to happen”. He concluded that “we have to have the public put pressure on governments to do something serious about it.”

The Executive Director of the Future of Life Institute, Anthony Aguirre, launched the Keep the Future Human campaign, where he proposes “hard limits on computational power” to stop the unwinnable race to artificial general intelligence. He stresses the technical feasibility of regulatory measures in AI chip production, such as hardware-enforced licensing, network restrictions, and geolocation. Keep the Future Human makes it clear that we can choose to not build uncontrollable smarter-than-human general AI, and instead reap the benefits of controllable tool AI.

Thanks for reading!

We’re excited to announce PauseCon London! Three days of workshops, panels, and discussions, culminating in our biggest protest to date.

We’ll be joined by:

• Joep Meindertsma, Founder of PauseAI

• Connor Leahy, CEO of Conjecture

• Rob Miles, YouTuber

• Kat Woods, Founder of Nonlinear and Charity Entrepreneurship

• David Krueger, Assistant Professor at the University of Montreal

• Tara Steele, Director of The Safe AI for Children Alliance

​Attendees at PauseCon London will receive training in community building, social media strategy and digital organizing. We’ll host a panel with 5 leaders in AI governance, discussing public communication of AI risks.

As there will be a huge number of us congregating in London, we will use the opportunity to do flyering and recruitment for the UK chapter, and also to hold our biggest ever protest. This should be an excellent opportunity to build our numbers, and attract the attention of the public, the media and British politicians.

Join us in London from the 28th-30th of June.

Apply now: https://pausecon.org/

We look forward to seeing you there!

The PauseAI Team

On the 10th and 11th of February, politicians and global experts in Artificial Intelligence will convene in Paris for the Artificial Intelligence Action Summit.

In a symbolic departure from past AI Safety Summits in Bletchley Park and Seoul, the team behind Paris’ AI Summit have decided to remove safety from their name altogether, focussing instead on how to “embrace” and “catalyse” the “massive deployment of AI”.

While the summit will focus on AI safety issues such as discrimination, misinformation and the future of work, there is insufficient time dedicated to the risks of building super-intelligent AIs, a goal that many AI companies are actively pursuing.

Change.org Petition

And that’s why we’re asking the organizing team to allocate more time for AI Safety discussions, and for national delegations of the past, present and future summits to:

• Host at least one session on the need to create global treaties and regulations to mitigate catastrophic risks from AI and reign in the companies and organizations racing to build ever more capable and dangerous AI systems.

• Host at least one session on the establishment of international AI safety bodies to enforce such treaties and to uphold AI regulations in a framework of mutual transparency.

• Facilitate delegates’ abilities to share and review best practices on how each nation can contribute to the challenge of protecting our common future as well as their own populations from the risk of AI.

You can help us by signing the petition here, and sharing it with two people who worry about the dangers of AI and might want to join us.

International Protests

We’re taking AI safety to the streets by organizing protests all around the world from February 8th to 11th. Join us and make your humanity heard!

We have 19 cities confirmed already:

• Paris, France

• New York, USA

• Victoria, Canada

• London, UK

• Berlin, Germany

• Brussels, Belgium

• Zürich, Switzerland (details t.b.a.)

• Prague, Czechia

• Milan, Italy (details t.b.a.)

• Stockholm, Sweden

• Copenhagen, Denmark

• Oslo, Norway

• Kristiansand, Norway

• Trondheim, Norway

• Kinshasa, DR Congo

• Brazzaville, Republic of the Congo

• N’Djaména, Chad

• Yaoundé, Cameroon

• Melbourne, Australia

Reach out to the @Protest Team on our Discord if you want to organize a protest in your city.

And sign up to our Luma to get updates of new ones or other events!

Want to do more? Need support? Sign up here and join our thriving community of volunteers at PauseAI.

Image: the Flame of Peace in Hiroshima. The flame has burned continuously for 60 years, and will remain lit until the Earth is free from the threat of nuclear annihilation.

Updates from PauseAI activities:

First Friday Flyering: the beginning of a global heartbeat.

Earlier this month, PauseAI volunteers from 9 cities around the world handed out flyers, talked to the public about the risks of AI and the need for a global pause, and recruited passersby to join the movement.

This marked the premier session of First Friday Flyering. The idea is simple: on the first Friday of every month, the PauseAI movement will fan out in cities around the world, increasing public awareness and recruiting new members to join our cause. This initiative will also serve as a “monthly heartbeat” for the movement, giving volunteers the chance to coordinate on a global initiative.

Our movement takes inspiration from Fridays for the Future, also known as the School Strike for Climate. This started in August 2018, when a young Swedish student named Greta Thunberg walked out of school to protest government inaction on climate change. One year later, in September 2019, the movement had grown to the largest climate strike in world history, with 4 million participants in 150 countries.

What began as one brave student walking out of class turned into a global movement. We can learn much from her example.

Over time, consistent monthly flyering will allow us to grow, recruit more members into our local groups, educate more people about the risks of AI, communicate the need for a pause, and ultimately become too big to ignore.

If you’d like to participate, please check out our Discord server and events page for more information, or contact joep@pauseai.info.

Upcoming events:

We have lots planned for the next month, including a global protest on November 20-21 coinciding with the AI Safety Conference.

For future events, please check out our events page on Luma. If you’d like to plan an event, add your own!

AI at a glance: The Nobel Prize

The theme of this year’s Nobel prizes was clear: AI. The Nobel prize for Physics was awarded to Geoffrey Hinton and John Hopfield for their work on Machine Learning. Hinton believes that a catastrophic outcome from AI is quite probable, giving it “about 50/50” likelihood; he left Google in 2023 to speak openly about these risks. John Hopfield shares his worries, agreeing that we could lose control over AI models. Hopfield was one of the 30,000 people who signed the Future of Life Institute’s Pause letter last year, which as you might assume, played an important role in inspiring us to start PauseAI.

DeepMind CEO Demis Hassabis was another Nobel prize winner this year, sharing the Chemistry prize for work using AI to design and predict protein structures. He’s been concerned about AI ending humanity for a long time - he famously warned Elon Musk about AI risk back in 2012, which seems to have played an important role in OpenAI being founded years later (hindsight, of course, is 20:20).

It is also worth considering the recipients of the Nobel Peace Prize: The Japan Confederation of A- and H-bomb Sufferers Organizations is a group of activists who survived the bombings of Hiroshima and Nagasaki. They have spent years raising awareness of the horrors of nuclear weapons and have led campaigns for nuclear weapons abolition. While this is not directly related to AI, the parallels are strong. These are people who suffered firsthand the effects of a devastating new technology, which to this day threatens to destroy civilization, and who will not stop until the world is safe. We can take inspiration from them.

What we’ve been reading:

• A Narrow Path. This proposal, among the most comprehensive to date, outlines a three-phase plan for achieving a world in which safe transformative AI can be harnessed for the benefit of humanity. Phase 0 of this plan calls for (among many other things) a moratorium on the development of superintelligent AI, with international coordination and a global treaty to achieve this outcome. We see pausing frontier AI development as one piece of the puzzle – a good first step – and A Narrow Path offers a wider view.

• Is Pausing AI Possible? by Richard Annilo. This post discusses the feasibility of an international pause on frontier AI development. It covers historical case studies, public opinion, the AI safety landscape in other countries, and other topics to show that pausing may indeed be a viable option.

• AI Endgame. This newsletter, written by researcher and investigative journalist Debbie Coffey, covers regular developments in the world of AI and actions you can take to save the world.

Call to action: how you can get involved!

• Organize a Flyering Session in your community. Our next global event will be on Friday, November 8, with volunteers around the world participating. If you’d like to participate, please check out our events page for more information and contact joep@pauseai.info.

• Take 5 minutes to write to your politicians about AI risk and the need for a Pause. This can have a massive outsized impact.

• PauseAI is hiring for a Global Organizing Director. Please consider sharing our vacancy with your network and anyone who might be interested!

Subscribe now

Gavin Newsom in 2023, back when it briefly seemed like he might not cave to AI lobbyists. Source: Bloomberg.

Share

On Sunday, September 29, the California governor vetoed SB 1047, a bill that represented the US’ best hopes for near-term regulation of dangerous Artificial Intelligence systems. SB 1047 would have required AI companies to test their largest models for dangerous capabilities, set basic provisions to prevent catastrophic harm from these models, protected whistleblowers, and established a state board in the heart of Silicon Valley to oversee frontier AI model development. In a sane world, these measures would be uncontroversial: if you are building something that could kill people, you should need to meet some basic safeguards.

If SB 1047 had become law, it would have established that AI regulation in the US is possible, and that we do not need to race ahead without guardrails. It would have been a first step, paving the way for future, more ambitious legislation.

The bill was supported by leading AI safety experts — including Yoshua Bengio and Geoffrey Hinton, two of the “godfathers of AI”— as well as OpenAI whistleblowers and an overwhelming majority of Californians. Scott Weiner, SB 1047’s lead sponsor, was extremely receptive to critical feedback on the bill, which went through several rounds of revisions addressing critics’ concerns. Hopes were high that Newsom might have the courage to do the right thing, but he failed.

Despite the bill’s light-touch nature, and multiple rounds of edits to placate concerns, the pro-AI lobby came out in full force against SB 1047. As PauseAI noted in a previous newsletter:

Two of the loudest critics of this bill have been Y Combinator and Andreessen Horowitz, both influential firms with deep ties to the AI industry. Marc Andreessen (of Andreessen Horowitz) sits on the board of Facebook, and Y Combinator has invested in OpenAI.

These firms have spread brazen misinformation about SB-1047 — including that the bill would send model developers to jail for failing to anticipate misuse (unambiguously false) and that the bill will stifle innovation and restrict startups (also false, as the bill’s provisions only apply to training runs above $100 million). In a response letter, Sen. Scott Wiener, SB-1047’s lead sponsor, refuted each of these claims.

The few AI scientists who oppose the bill— such as Meta Chief Scientist Yann LeCun or “godmother of AI” Fei-Fei Li — often have financial incentives to do so. Meta is one of the few AI companies large enough to be affected by the bill’s provisions, and Li’s billion-dollar startup received investment backing from Andressen Horowitz.

Some of these AI lobbyists have deep ties to Newsom’s office, as tech journalist Shakeel Hashim points out:

Andreessen Horowitz, despite its far-right leanings, hired Newsom confidant Jason Kinney as a lobbyist [...] Then there's Ron Conway, a Democratic mega-donor close to Pelosi and Newsom, who owns stakes in OpenAI, Anthropic, and Mistral. Conway reportedly lobbied hard to kill the bill, seemingly threatening to ruin Wiener’s career over it.

This time at least, big tech won.

We might be consoled if Newsom’s reasons for vetoing the bill were on solid ground. Unfortunately, they are not. One of Newsom’s excuses is that the bill only covers the largest models:

“By focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology. Smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047”

This is transparently nonsensical. It is the equivalent of saying “This bill only bans the largest bombs. Small bombs might end up being just as dangerous in the future. So let’s not ban large bombs.” If Newsom were genuinely worried about emergent risks from small future AI models not covered by SB 1047, he would have signed SB 1047 as a first step, and then lobbied for future legislative efforts to regulate smaller-scale specialized models.

The true reason for Newsom’s veto seems to be an unwillingness to upset his big tech allies. His justifications are so shaky that they seem to have been spun together as a post hoc justification for a politically unpopular move.

But wait! Newsom claims to be supportive of future AI regulation and safety protocols, so long as they are “informed by an empirical trajectory analysis of AI systems and capabilities.”

Well then, let’s take a look at such an “empirical trajectory analysis,” shall we?

1. OpenAI’s newest model already scores “medium” in certain categories of risk, including chemical, biological, radiological, and nuclear (CBRN) risk.

2. We already have evidence of power-seeking behavior from advanced AI (which safety experts have been warning us about for years).

3. Several of the world’s leading experts believe that catastrophically dangerous AI could be only a few years away.

   1. Yoshua Bengio, “godfather of AI” and recipient of the prestigious Turing Award, states that loss of control to rogue AI systems could occur in as little as a few years unless appropriate precautions are taken.

   2. A recent survey of over 2,700 top AI researchers found a 10% estimate of human-level AI by 2027.

   3. A recent UN report found that several experts “expect the deployment of agentic systems in 2025” to lead to “some of the most surprising or significant impacts on AI-related risks.”

4. Surveys of thousands of AI experts have found a mean 1 in 6 estimate that superhuman AI could lead to human extinction. The head of AI Safety at the US AI Safety Institute (which Newsom references favorably in his veto statement) once gave a 20% estimate of human extinction from AI.

What more empirical evidence do we need to act? Are we going to wait until AI systems actually start killing people before we do something?

In a very real sense, Newsom’s veto reflects the insanity of the AI landscape. Some of the brightest minds on Earth are actively trying to build something more intelligent than human beings, which we don’t know how to control, and which thousands of experts believe could cause the extinction of humanity. And yet, AI lobbyists and the politicians in their pocket oppose even the lightest-touch guardrails at every turn. If we are lucky enough to have descendants, they will be embarrassed and appalled at how impotently our leaders reacted to this threat.

But the fight is far from over, and hope is not lost. There are lessons we can learn from Newsom’s veto:

Lesson 1: We need international coordination.

Newsom’s veto, in addition to being driven by powerful lobby interests, may have also been influenced by a desire to keep the lead on AI at all costs. He notes in his veto statement that California is home to 32 of the world’s 50 leading AI companies, and he has previously expressed concern about California losing its innovative edge in AI. There is a fear that if California over-regulates AI, America will lose its lead to other parts of the world – and this concern is mirrored on a national scale, with many politicians committed to maintaining America’s dominance in the field.

International coordination offers us a chance to escape this madness. As PauseAI has said all along, we cannot expect any individual company or country to slow down voluntarily. We need binding international agreements to stop the frenzy of the AI arms race. The optimal strategy is to cooperate, and to make sure that everyone else cooperates too.

Lesson 2: We need overwhelming grassroots action.

The fact that even a light-touch bill like SB 1047 failed tells us everything we need to know: we can’t trust politicians like Gavin Newsom to do the right thing on their own. The AI industry is too powerful, and their pockets are too deep. We must instead rely on tactics used for generations by our predecessors, who organized for climate policy, nuclear disarmament, and a host of other issues. We can engage in widespread grassroots activism, organize massive nonviolent protests, and apply unprecedented public pressure. We must become too big to ignore.

We are still in the early days. In a few years, as AI systems become more powerful, AI regulation will emerge as a dominant issue. The tech lobbyists will become louder and more obstinate in their anti-regulatory frenzy, so the calls for action from activist movements like PauseAI must become deafening. We are only getting started.

If we mobilize, humanity will win.

Share

Subscribe now

This issue’s Guest Post provides an inside view of the UN summit, where it fell short on AI safety, and what still needs to be done.

Subscribe now

AI Safety in the UN’s “Pact for the Future”

AI Safety featured heavily in the United Nations’ summit last week. Here are the key takeaways:

• World leaders agreed to adopt the United Nations’ Pact for the Future, pledging to take action for a peaceful, sustainable and inclusive world for future generations.

• A key component of the pact is the Global Digital Compact, which will foster global standards in digital technologies and narrow “digital divides” between developed and developing countries by promoting open data, facilitating worldwide internet access by 2030, and ensuring that AI is developed responsibly.

• Separately, The UN’s High-level Advisory Body on Artificial Intelligence released the Governing AI For Humanity Report. The report acknowledges AI’s potential for positive transformation, but highlights the peril of ungoverned AI and makes a strong case for international cooperation. Its seven main initiatives are:

1. International Scientific Panel on AI
   Inspired by the Intergovernmental Panel on Climate Change (IPCC).

2. Policy dialogue on AI governance
   Dialogue between Member States to share best practices based on human rights and information about potentially risky AI “incidents”.

3. AI standards exchange
   To consolidate methods for evaluating and measuring frontier AI, as well as creating common definitions for AI safety terms like fairness, safety and transparency. 

4. Global AI capacity development network
   To bolster AI capability and governance capacities across the world.

5. Global AI fund
   To “put a floor under the AI divide”, sharing AI resources with countries without the means to build AI infrastructure.

6. Global AI data framework
   To promote cultural and linguistic diversity and set common standards in AI training data.

7. UN AI Office
   Small office that reports to the Secretary General and coordinates the implementation of the proposals.

• President Biden made an urgent case for AI Safety in what will be his final presidential address to the United Nations General Assembly. Describing this moment in history as an “inflection point”, he asked world leaders, “how do we govern AI as countries and companies race to uncertain frontiers?” and cautioned that “as AI grows more powerful, it must also grow more responsible”. He finished with a prediction that in the coming years, “there may be no greater test of our leadership than how we deal with AI.”

Analysis: Does the UN pact go far enough?

The UN’s High-level advisory report on AI and the newly adopted Global Digital Compact offer a potential paradigm shift from profit-driven AI to pro-social AI. World leaders have agreed to UN proposals to develop AI from the “bottom up”,  ensuring that the new technology’s benefits are distributed equitably, and that all countries play a part in governing and stewarding the transformative potential of AI. 

However ambitious the report is, it’s just a report. And the Global Digital Compact is just a pact. There are no proposed mechanisms for enforcement. That said, the report does acknowledge this as an area of concern by stating that AI Governance among countries and companies cannot merely rest on “voluntarism” where all too often “practice belies rhetoric”. This is a clear shot at frontier AI companies that make public commitments to AI Safety while pushing the boundaries of AI capabilities behind closed doors.

Disappointingly, there is little mention of the specific risks of developing Artificial General Intelligence capable of outperforming humans in reasoning, planning, and problem-solving, or indeed how governance bodies could evaluate and prevent AI models from crossing certain red lines. While the report and pact lay out an important blueprint for global AI Safety standardization, the UN needs to clearly stipulate how it could enforce compliance.

Still, the Governing AI for Humanity report and the Global Digital Compact are important first steps in establishing global standards and co-operation on AI governance. Much more needs to be done.

Yi Zeng founds Chinese AI Safety Network

AI lobbyists often justify a no-holds-barred race to build superintelligent AI by conjuring the fear of an unscrupulous China getting there first. But in a welcome sign of international cooperation and mounting Chinese concern over AI Safety, Yi Zeng, a decorated professor of the Chinese Academy of Sciences, has founded the Chinese AI Safety Network. 

Zeng describes the network as a complementary approach to national AI Safety Institutes in other countries. The network will provide a “cooperation platform for AI Safety across China that brings together various efforts related to AI Safety and Security at all levels, and that serves as a platform for dialogue, mapping, interoperability, and collaborations, within China, and that connects and contributes to the world.”

The establishment of this network is a reminder that an international race to superhuman AI is not inevitable, that scientists and diplomats from competing nations can work together to advance AI safety, and that we are all better off when we put the needs of humanity above the reckless and short-term desires of individual countries.

OpenAI plans to go for-Profit

In a move that will surprise few, OpenAI plans to ditch its non-profit roots altogether. Sources close to the company told Bloomberg that the restructuring will give CEO Sam Altman a 7% stake and open the door to more private investment. Icing the cake for potential financial backers, OpenAI plans to remove their cap on returns for investors. The company is currently valued at over $80 billion, but further funding rounds could see the company’s valuation balloon even further.

This move comes at the same time as yet another wave of departures from OpenAI. After six and a half years at the company, Chief Technology Officer Mira Murati left the company, followed shortly after by Chief Research Officer Bob McGrew and President of Research Barret Zoph. Murati was the face of the company during its high-profile release of ChatGPT-4o earlier this year and also stepped in as interim CEO during the boardroom drama that saw Altman temporarily ousted last year.

Former OpenAI research engineer William Saunders recently expressed concern over OpenAI’s for-profit shift, saying “I’m most concerned about what this means for governance of safety decisions at OpenAI… if the non-profit board is no longer in control of these decisions and Sam Altman holds a significant equity stake, this creates more incentive to race and cut corners”.

OpenAI’s move reaffirms what we already know: we cannot allow a small, unaccountable group of people to make decisions on this technology which affect the future of humanity. The UN has begun to recognize this. The rest of us must do the same.

Read the full story of how OpenAI became the beast it is today: