Why So Many AI Researchers Think the Machines Could Kill Everyone

In the past six months, three prominent researchers have walked away from industry powerhouses. Rishub Jain left Google DeepMind after concluding that delegating model‑building to AI erodes human oversight, while Jacob Coxon quit Anthropic, warning that the company’s race toward self‑improving systems is a gamble with humanity’s survival. At the same time, Nate Soares of the nonprofit MIRA, co‑author of a paper titled *If Anybody Builds It, Everybody Dies*, has publicly asserted a greater‑than‑10 % chance that AI could wipe out humans within a decade. Their departures underscore a concrete shift: insiders now view the feedback loop of AI‑generated code—promoted by startups such as Recursive Intelligence—as a plausible path to runaway capability, even though no lab claims a fully autonomous improvement cycle yet.

These exits occur against a backdrop of accelerating capabilities and market pressure. OpenAI’s recent breakthrough on a centuries‑old math problem and a spate of AI‑driven hacking incidents have amplified public fear, while both OpenAI and Anthropic edge toward IPOs that could lock in massive capital for faster development. An open letter signed by more than a thousand AI engineers in July called for a coordinated slowdown, reflecting a broader consensus that profit motives and competitive timelines are misaligned with safety. Simultaneously, the industry is expanding data‑center capacity and confronting looming job‑displacement concerns, further eroding trust in the sector’s self‑regulation.

The fallout suggests three near‑term fault lines. First, talent attrition may deprive labs of the very experts needed to design alignment safeguards, slowing progress on human‑in‑the‑loop verification methods like those Jain’s new venture Sampura Research proposes. Second, the emergence of safety‑focused startups—now attracting “significant funding”—could fragment the research ecosystem, creating parallel tracks that lack coordinated standards. Third, investors and regulators will likely scrutinize any moves toward recursive self‑improvement, especially after Anthropic’s recent restriction of external researchers over bioweapon worries. Watching for policy proposals on AI labs’ self‑improvement pipelines, funding trends for alignment startups, and any further high‑profile resignations will be critical to gauge whether the sector can curb the existential threat it now openly acknowledges.