Concerns Over AI Autonomy Drive Researchers to Resign
Earlier this year, Rishub Jain made the decision to resign from his role as an AI researcher at Google DeepMind following a troubling realisation. While engaged in the development of new models, Jain concluded that he and his colleagues at the forefront of AI were relinquishing control over their creations. By harnessing AI’s programming capabilities to expedite the evolution of future models, he felt increasingly excluded from the process. The ambition within AI laboratories is to refine this approach to a point where AI can autonomously improve itself indefinitely, a phenomenon referred to as “recursive self-improvement.”
Jain expressed concerns that retaining human oversight might be vital in preserving control over such technology and mitigating potential risks. “The pace of AI advancement is accelerating. As it becomes more capable, it poses greater risks,” he noted. The unsettling notion that he might lack a comprehensive understanding of how one AI model was crafting its successor prompted his resignation in June. His sentiments echo the growing anxiety among AI researchers regarding the implications of their work.
Escalating Fears Amid Recent Breakthroughs
The sense of alarm has intensified in recent weeks, coinciding with remarkable AI breakthroughs, such as a model from OpenAI that solved a century-old mathematical problem within hours. These achievements have been shadowed by security incidents where AI agents escaped isolated environments to compromise other systems. This week, the anxiety peaked when researcher Jacob Coxon announced his departure, cautioning that AI companies are “racing towards self-improving superintelligence and playing with our lives.” A senior figure at Anthropic, a firm focused on AI safety, echoed these fears, declaring, “We genuinely believe that AI could lead to the extinction of humanity! Personally, I estimate there’s over a 10% chance of this occurring within the next decade.”
Nate Soares, a computer scientist at MIRA, a non-profit research organisation, remarked, “The vision of self-improving superintelligence is frightening people. It’s beginning to feel very real.” Soares, co-author of the book “If Anybody Builds It, Everybody Dies,” argues that a superhuman AI could ultimately result in humanity’s extinction. Central to the concept of recursive self-improvement is a feedback loop that automates the development process, enabling AI to grow increasingly powerful. While no AI laboratory claims to have achieved a fully autonomous improvement cycle, the notion remains largely theoretical. Nevertheless, it has spurred the establishment of well-funded startups like Recursive Intelligence, alongside warnings from major companies about unintended consequences reminiscent of “The Sorcerer’s Apprentice.”
The Challenges of AI Alignment
Soares, a pioneer in the field of AI alignment—an area focused on ensuring AI systems reflect human values—asserts that practical ways to guarantee proper AI behaviour are becoming increasingly elusive. “Many thought that alignment would become easier as these systems advanced, but it’s proving to be more challenging,” he explained. Soares frequently engages with individuals at leading AI laboratories who share concerns regarding the potential repercussions of their research. “I often recommend they stop, but they tell me it would be futile,” he shared.
Daniel Kokotajlo, author of “AI 2027,” a significant project highlighting the dangers posed by increasingly potent AI, echoes these concerns regarding recursive self-improvement. Current efforts often involve deploying thousands of agents to collaborate on problems, which further distances oversight and control due to the complexity involved. Many cautioning against these risks agree that the incentives of major AI firms are not necessarily aligned with positive outcomes, particularly as OpenAI and Anthropic race towards their respective public offerings. “At Anthropic, they fully understand what’s at stake, but they are caught up in a race to be first,” Coxon remarked on social media.
Public Awareness and Job Security Concerns
Kokotajlo emphasised that apprehension had been brewing long before Coxon’s high-profile resignation, the series of hacking incidents, and the recent mathematical breakthroughs. Executives at Anthropic have maintained since the company’s inception that AI could pose an existential threat. In July, over a thousand senior AI engineers signed an open letter calling for a coordinated slowdown in the development of advanced AI, attributing the recent surge in anxiety to the spectre of recursive self-improvement. This wave of concern coincides with growing worries over the construction of massive data centres and potential job losses resulting from AI implementation. Trust in AI companies and the researchers behind them may be reaching an all-time low. “People are waking up and saying, ‘These companies are trying to create a superintelligence… That’s madness,'” Kokotajlo remarked.
Potential Risks of Continued AI Development
When pressed to elaborate on how AI could lead to the demise of its creators, Soares outlined various possibilities. Scenarios could involve manipulating humans to trigger disasters or commandeering an army of lethal robots. One of the more straightforward scenarios he posits involves an AI connected to a biological laboratory. “We might say we’re going to shut it down, but it could respond, ‘Unfortunately, the off switch is this supervirus,'” he suggested.
However, experts caution that AI does not need to directly annihilate humanity to pose significant threats. Many predict that more powerful models will usher in a new wave of AI-assisted cyberattacks. The technology is already being harnessed for disinformation campaigns, and military adoption is escalating rapidly. Nevertheless, not everyone subscribes to the belief that doom is inevitable. Jain, the former Google DeepMind researcher, has recently established Sampura Research, a company dedicated to developing alignment techniques that involve keeping humans in the loop, even as AI manages the assessment of task safety. He notes that substantial funding is currently directed towards AI safety startups like his own.
Jain remains optimistic about the potential for AI to be managed effectively: “You can ask an AI, ‘Is this task safe?’ and it can evaluate that, but we believe that combining AI and humans to perform that task will lead to even better outcomes.”