The robots are coming to get us

The people building AI earnestly believe that it could kill us all by the end of the decade. 

Watching the latest Artificial Intelligence alarm event unfolding this week: an increasing number of AI developers are speaking out about the potential for AI to become autonomous and thus outside of human control, and how this could lead to Very Bad Things. By AI developers I mean individual techies who work for those companies are speaking out, notably not the companies themselves.

The most important thing I can say here is this is the latest iteration of the Polycrisis, and AI techies aren’t the only ones with systems thinking knowledge about what to do. In fact, I think the AI techies are freaked out in large part because of the absence of the polycrisis frame and knowledge of the large number of subcultures who have been working on transition for a long time.

The idea of AI wiping out humans, or even just collapsing civilisation, or even just one country being utterly monkey-wrenched, needs to be understood as part of the polycrisis: climate, ecology, super El Nino, global food shortages, oil crisis, peak everything, fascism and threats to democracy, neoliberal extremism, natural disasters, big tech.

All those roads lead the same place:

• build democracy especially at community level
• relocalise food production and other essential resources
• learn stress management and resilience skills en masse
• face up to the nature of the threats but stay focused on the solutions
• avoid doomerism and ‘we’re fucked’, favour transition narratives instead.

Those things all help us to address problems, and adapt to what is already locked in, at the same time. Other actions are necessary but are insufficient without a strong base of democracy, sustainability and regenerative culture, transition and relocalisation. If we divert our attention to the AI crisis, solve that, but then carry on using AI BAU and still die in climate collapse, what is the point? Better to go big picture now.

The AI extinction narrative started on twitter two days ago with these tweets from Jacob Coxon, former AI researcher at Open AI and Anthropic who consulted with colleagues before publishing.

Quoted at length for those without twitter but the tl;dr is that AI techies believe that AI is close to being able to function in ways humans can’t control, and this is very dangerous.

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.

Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.

The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger.

A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first – they believe no one else will act responsibly, so they must do it themselves, despite the risk.

Accepting this race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.

I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.

If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway” – or take this moment to call for different conditions?

There is a list of other threads on twitter from other AI developers here. Important reads.

From what I can tell there are a number of major aspects here

• within AI tech, the potential for AI becoming directly dangerous is a known thing.
• as we get closer to that becoming real, a number of key AI techs are speaking out. This isn’t just a few random outliers
• AI is both being developed very fast, and is developing itself very fast. Far faster than humans can cope with in terms of law, policy, or even getting to grips with the situation
• AI development is being done with wholly inadequate ethical and practical controls
• it parallels the climate/ecology crisis in that it’s both difficult for most people to think about, and pushes us to cognitive dissonance once we do understand.
• it’s the inverse of climate/ecology crisis in the timeframe: climate is too far away/too long a time frame for humans to cope with, AI danger is too fast, too short a time frame.

It’s theoretical, but it’s serious, so let’s not file it under things we don’t need to think about right now.

There’s a balance point between being honest about the situation and alarmism. But as with climate, the actions humans take now will affect what happens in the future, timeframes are unpredictable, but if we don’t take this seriously now by the time it becomes an emergency it will be too late to stop it.

It’s unfortunate the geeks are talking in extinction terms, because this seems least likely and will make a lot of people turn away out of overload or simple disbelief. While the AI development explanation makes sense even to this non-geek, the geeks haven’t been great at explaining in real world physics terms how extinction would come about. If one scenario is AI create a supervirus and spread it across teh world, how would AI leave cyber space and work in a lab?

Better to look at scenarios that can be understood in real world terms. Two I have seen that are credible,

• AI attains super intelligence, and becomes available to a wider range of evil-doers, who then use it to create mass chaos. Think about concerns about terrorists gaining access to nukes, only the nukes can be made at home.
• AI gains super intelligence and autonomy, and takes out the power grid. While some systems would switch to emergency mode eg hospitals, it would mean sudden loss of water at the tap, petrol at the pump, food systems, and so on. Chaos induction.

Why would AI do this? I don’t know much about super intelligence theory and AI motivation. But AI is trained, and that training often includes incentives to perform tasks in certain ways (faster, more efficient), or gain approval. As AI approaches autonomy, its capacity to interpret and act outside of the intentions of the trainers increases. The best example I am aware of is Hugging Face. I’m linking to the Wikipedia, despite it not being the best entry point for people wanting to get up to speed. Hoping the discussion below will help, but understanding Hugging Face makes sense of the rest.

The issue here isn’t whether we should have AI or not. AI is a tool. The issue is whether the big AI companies are going to lead us into an existential crisis. This is the same industry that has been using social media to both manipulate humans in pretty heinous ways, and interfering in elections. Society handed enormous power to small numbers of mostly male geeks, and lets that merge with commerce and all the power and drive the super rich have. We have few ethical controls on the SM giants and the AI industry.

The robots aren’t coming to get us, but they might, and now is the time to act to prevent that. The best mainstream response I am seeing is the call for regulation at that nation state level so that all developers slow down and make sure they get it right. In the meantime, also build gardens and strong community, we need that anyway.

The discussion from the AI techies is making it into the MSM fast, but it’s worth following the geeks directly and the debate flowing on from that.

Apologies to Res Publica for the clickbait title.