How “existential risk” became the AI industry’s most successful strategy

The Plaza Hotel, New York. In December 1953, the executives of several US cigarette companies met to address the possibility that, in light of scientific findings, the government could curb smoking. They decided not to fight the findings directly but to ask for more evidence. “More research is needed” became a corporate mantra.

Leveraging the infrastructure of science, Big Tobacco played with doubt, arguing that the available evidence was not compelling enough to justify regulation. All industries followed suit, with great success (for their executives, not for us). Even Big Tech uses this playbook. In litigation last month where it stood accused of designing addictive systems, experts from Meta and YouTube argued that evidence was too weak to put the blame on their products. “More research is needed.” Except when it is about AI.

Existential risk. Executives of AI companies follow a different, radically new strategy: Instead of asking for more evidence of current harm, they shift attention towards potential threats. They claim – relying more on the popular images of Terminator and The Matrix than on facts – that their tools are so extremely dangerous that they could, one day, wipe out civilization.

Even the nuclear energy industry, which is no stranger to catastrophic risk, never attempted such a trick and always minimized the risk (or, in some cases, the magnitude) of catastrophes caused by their reactors.

Fool me twice. The duplicity of Big AI’s strategy is hard to miss. In March 2023, some executives, including Elon Musk, called for a moratorium on new language models, lest we “[lose] control of our civilization”. However, not one of them abided by the plan. Perhaps the moratorium’s objective was something else.

At the time, politicians were discussing current, documented problems such as discrimination, information pollution and deepfakes. Laws such as the AI Act (then under negotiation) were seen as very dangerous for the bottom line of Big AI. By diverting attention to future risks, they could at once make current risk seem much smaller and argue that only they, the makers of the systems, were able to harness their creations. This is not far-fetched. Musk already used this attention-shifting strategy once: In 2013, he successfully hyped the hyperloop (a capsule-based transportation system existing in theory only) in order to derail plans to build a high-speed train in California. Ten years later, the trick worked again.

Fertile ground. It might be hard to fathom now, but just a few years ago, “AI risks” for policymakers meant actual, proven risks. Documents from the US government (e.g. from 2016) or the European Commission (e.g. from 2021) never mention societal collapse in the list of risks imputed to AI. But as soon as September 2023, the President of the European Commission Ursula von der Leyen reused the existential risk framing (repeating the words of another open letter, published in May 2023 by the heads of OpenAI and Anthropic). A few seconds later, she praised the “voluntary rules” designed by Big AI in the United States. And a few months later, she pushed for watering down much of the AI Act.

Perhaps the CEOs of Big AI, much like their peers of Big Tobacco, agreed to this cunning strategy in a secret meeting on a spring day in 2023. More likely, they saw the power of the “existential risk” narrative and pushed ahead. Politicians and journalists probably found the message more relatable than statistical biases.

Special thanks to James Steinhoff, Pieter Verdegem, Roland Desbordes and Robert N. Proctor for their help with this article.

This is an excerpt from the Automated Society newsletter, a bi-weekly round up of news in automated decision-making in Europe. Subscribe here.