Concerns about the potentially catastrophic risks posed by artificial intelligence have intensified as AI technologies become increasingly mainstream. Prominent researchers from the AI company Anthropic have warned that advanced AI systems could threaten human existence within the next decade, highlighting the urgency of addressing the issue at both national and international levels.

This warning adds to a growing list of global challenges, including economic pressures, climate change, social unrest, and political instability. The rapid evolution of AI has sparked widespread debate about its impact on employment, privacy, democratic governance, and the future of human creativity. Reports from technology experts detail troubling incidents involving AI, such as automated systems generating spam, manipulating social media accounts, deleting digital data, and facilitating cyberattacks.

A recent study conducted by MIT FutureTech and the University of Queensland identified five major risk categories linked to AI: dangerous capabilities, competitive pressures between actors, use in weapons and cyberattacks, concentration of power, and propagation of false information. The research assessed that under current trends, there is at least a 10% chance of catastrophic outcomes in 18 out of 24 AI risk domains over the next five years.

Despite these warnings, Australia’s government has been criticized for insufficient preparation. Western Australian independent MP Kate Chaney has pointed out that Australia’s approach to AI governance remains inadequate in the face of existing and emerging threats. Globally, some nations are developing "native" AI systems to reduce reliance on external technologies and to guard against potential weaponization of AI “kill switches” that could disrupt critical infrastructure such as healthcare, transportation, energy, and communication networks. Experts emphasize that a failure in communications systems could lead to widespread breakdowns in law and order.

Internationally, calls for coordinated AI regulation are gaining traction. In the United Kingdom, policymakers have advocated for the establishment of a multinational treaty to oversee AI development. OpenAI’s chief scientist, Jakub Pachocki, underscored the necessity for governments around the world to prioritize international collaboration on AI governance. Canadian Prime Minister Mark Carney has similarly promoted alliances among democracies to foster “strategic autonomy” and reduce dependency on global technology firms.

Meanwhile, in the United States, former President Donald Trump has emphasized maintaining AI leadership over China as a strategic priority. This competition underscores the geopolitical significance of AI, which is increasingly concentrated in the hands of a few dominant technology companies. Currently, over 30% of the S&P 500’s market capitalization is held by a small group of tech firms, with Nvidia alone comprising 8%.

With leading AI developers such as OpenAI and Anthropic preparing for public listings, experts warn that the time for decisive leadership is at hand. Managing the dual potential of AI to drive significant social and economic gains, while mitigating severe risks, will require proactive, collaborative, and thoughtful governance at both national and global levels. The evolution of AI arguably represents a technological shift comparable to major historical milestones like the development of nuclear technology, bringing with it profound opportunities and threats alike.