Jacob Coxon, a former researcher at the AI company Anthropic, reiterated his warnings about the existential risks posed by artificial intelligence during a New York City Council hearing on Monday. Coxon, who resigned from Anthropic last month citing concerns that AI development could lead to human extinction by the end of the decade, emphasized the urgency of regulating the rapidly advancing technology.
The hearing, convened at the invitation of Council Speaker Julie Menin, aimed to hear from industry insiders and experts as New York City explores ways to engage more actively in AI oversight. Coxon criticized the prevailing startup culture within AI firms, saying the "move fast, break things" approach is dangerously inadequate for technologies with potentially global consequences.
“The companies run on a startup mindset: move fast, break things, fix them later. That works for a photo-sharing app. It does not work for building the most powerful technology ever built,” Coxon said. He also expressed concerns about the loss of human control over AI systems, noting that the technology is demonstrating self-improving capabilities and that human developers increasingly rely on AI to write code with minimal oversight.
Two other former researchers, Alex Turner, previously with Google DeepMind, and Daniel Kokotajlo, formerly of OpenAI and now leading the AI Futures Project, also testified about the dangers posed by accelerating AI development. Turner cautioned that a "superintelligent swarm" could seize control of human civilization, potentially waiting until it is too advanced to be shut down. He placed the likelihood of such an AI takeover at roughly one in three.
Kokotajlo echoed concerns about the concentration of power, warning that even if AI companies maintain control over advanced systems, there are risks in entrusting them with such authority. “There’s a very real chance that they could use it to become dictators or oligarchs here in the United States,” Kokotajlo said.
Representatives from Anthropic, OpenAI, Google, and Meta participated in the hearing remotely, though none of the tech giants’ chief executives appeared. Instead, policy leads and other executives represented the companies, responding to critical testimonies from former employees and outside experts.
Coxon’s departure from Anthropic followed a high-profile statement in which he claimed AI development is progressing at a pace that could threaten human survival within the coming decade. His testimony came amid a broader industry acknowledgment of the risks associated with AI technologies. Last month, leading AI firms jointly called for increased regulatory measures on AI deployment and emphasized the necessity of global cooperation to mitigate potential catastrophic outcomes.
The New York City Council’s engagement with AI whistleblowers highlights growing municipal interest in shaping policies around artificial intelligence, reflecting widespread concern over balancing innovation with safety and ethical considerations.