Why Tech CEOs Are Suddenly Talking About Human Extinction

Tech CEOs discussed growth a few years ago. They discussed the next billion people coming online, monthly active users, and cloud margins. They are now discussing extinction.

It’s a subtle but noticeable change. Executives creating the most potent AI systems in the world have started drawing comparisons between their products and pandemics and nuclear weapons on podcasts, in Senate hearings, and in well-crafted open letters. According to a 2023 statement signed by Sam Altman and other prominent figures in the industry, “reducing the risk of extinction from AI should be a global priority.” Slipping that heavy word into a press cycle is a tough task.

As this is happening, it seems like something in Silicon Valley has become introspective, almost nervous.

Consider Altman. Long before ChatGPT was a household name, he wrote blog posts in the past about superintelligence posing serious threats to humanity. It sounded theoretical, almost academic, at the time. These days, search engines, productivity software, and even government tools are incorporating OpenAI’s models. Now, the alerts come in while the systems are already operational, servers whirring in massive data centers outside of Dublin and Dallas.

It’s difficult to overlook that tension. The chorus has been joined by other leaders. Statements putting AI at risk of extinction alongside nuclear war have been publicly endorsed by executives from Microsoft, Anthropic, and Google DeepMind. The likelihood of disastrous AI outcomes is “pretty high,” according to Sundar Pichai, who recently expressed hope that humanity will band together to avert catastrophe.

Or perhaps this is just intellectual integrity. Ultimately, it would be irresponsible to overlook the worst-case scenarios when developing systems that have the potential to surpass humans in numerous domains in the future.

However, there is an alternative interpretation. Tech leaders raise the discussion beyond commonplace negatives like bias, false information, and job displacement and into the domain of philosophy and survival by presenting AI as an existential threat. It shifts the debate’s balance. Only international coalitions and top-level regulatory agencies can act if extinction is the threat. This conveniently ignores local regulators and smaller critics.

Apocalypse seems to be evolving into a branding tactic. Earlier this year, computer scientist Stuart Russell warned of a potential “arms race” that could wipe out humanity at an AI summit in New Delhi. Young engineers gathered around espresso machines outside the convention center, exchanging notes about venture funding and model performance. It was a startling contrast. Existential dread is inside. Recruiting pitches outside.

It’s difficult to overlook the fact that these extinction warnings frequently accompany proposals for regulations that would strengthen current players. Trillion-dollar corporations can handle licensing laws, global oversight organizations, and extensive compliance frameworks. For startups using shared office space in San Francisco’s SoMa district, they are more difficult to manage.

In the meantime, AI’s immediate effects are already apparent. Concerns about the automation of customer service positions have caused outsourcing companies’ share prices in India to decline. Writers in Hollywood objected to studios testing scripts created by artificial intelligence. In an effort to identify essays that were generated by a computer, teachers are covertly revising assignments. You don’t need superintelligence for any of that. It needs software that is extremely powerful.

Opponents contend that dwelling on extinction is a diversion from these real-world disturbances.

Princeton computer scientist Arvind Narayanan has noted that existing AI systems are far from being able to plan apocalyptic events. Others are more concerned about what one researcher dubbed “fracturing reality”—AI systems that subtly influence elections and undermine public trust by saturating social media with credible false information. That future seems more real, less dramatic, and possibly more unstable. The story of extinction, however, endures.

Maybe some of it is real fear. Decades have been spent by many AI researchers envisioning systems that are more intelligent than humans. They might sense that things are changing as they watch models get better every year. Geoffrey Hinton appeared less like a marketer and more like a man uneasy with his own invention when he quit Google to talk more openly about the dangers of artificial intelligence.

However, tech CEOs may also be controlling expectations. They present themselves as responsible stewards rather than careless builders by admitting that there are significant risks. Even as capital investments in AI infrastructure reach the hundreds of billions, it gives the appearance of careful guardianship. Notably, investors have not run away.

Markets are still rewarding businesses that invest in generative AI. Energy demand is increasing, data centers are growing, and new models are being introduced with joyous livestreams. The CEOs who are threatening extinction are also praising enterprise adoption rates and productivity increases.

The conflicting messages of caution for humanity and optimism for shareholders can seem incongruous.

A more profound cultural change is at work. For many years, technology was promoted as a clear-cut benefit that would speed up development, connect people, and democratize information. Even its architects now use more somber language. Geopolitical instability, pandemics, and climate change have all undermined faith in linear progress. AI absorbs the anxiety as it arrives in the midst of that uncertainty. Whether extinction talk is a form of Silicon Valley melodrama or realistic foresight is still up for debate.

It’s simple to envision two futures playing out simultaneously while standing outside a brand-new data center and listening to the steady mechanical drone. In one, AI systems unlock scientific discoveries, optimize energy grids, and speed up drug discovery. On the other hand, misaligned systems intensify human conflict or deviate from our control in ways that are difficult to comprehend.

It appears that the executives creating these systems are torn between those two visions.

Why are tech CEOs now discussing the extinction of humans? Perhaps because it seems reckless to ignore the possibility now that the stakes are so high. Perhaps because the regulatory landscape is altered when existential risk is invoked. Or perhaps it’s because the tools they’re developing feel more like something unpredictable than software for the first time.

The language has evolved, regardless of the motivation. It’s important to listen to the people who are creating the machines of the future when they start using apocalyptic language, not only to what they are afraid of but also to what they are still creating.