The artificial intelligence industry was caught entirely off guard during the summer when high-profile employees at some of the world’s leading AI labs began publicly debating whether their own commercial products could ultimately trigger human extinction. The public reckoning began when Anthropic researcher Jacob Coxon announced his resignation via social media, issuing a stark warning that his former employer and rival OpenAI were "racing straight to self-improving superintelligence and gambling with our lives."
The internal alarms were immediately validated when Evan Hubinger, Anthropic’s head of alignment science, replied publicly to the post, stating, "Jacob is correct here—we really do earnestly believe AI could kill all humans!" Shortly afterward, Anthropic CEO Dario Amodei reinforced those sentiments in an interview with CNN, declaring that he agreed with Coxon far more than he disagreed.
These existential anxieties are rooted in a foundational hypothesis that has circulated within futurist circles for decades: once software engineers successfully design an AI system capable of independently improving its own code and architecture, it will ignite an accelerating cycle of autonomous upgrades known as recursive self-improvement. The ultimate destination of this loop is artificial superintelligence, or ASI, a hypothetical entity possessing cognitive capabilities so vastly superior to those of biological humans that it would effortlessly evade human control and potentially eradicate the human species entirely. Achieving artificial superintelligence represents a vastly more ambitious and perilous milestone than artificial general intelligence, which broadly defines systems capable of matching human performance across a diverse array of cognitive and practical tasks.
The tech sector and the public alike have reacted to these claims with deep polarization. Skeptics argue that predictions of impending ASI are frequently leveraged as marketing tools or as strategic maneuvers to encourage burdensome regulations that help incumbent labs squash emerging competitors. Other observers focus heavily on the theoretical plausibility of rogue systems destroying humanity, intensely debating the mechanics of hyper-specific scenarios like AI-engineered superviruses. Meanwhile, techno-optimists insist that aligning ASI with human interests is entirely feasible, urging society to rush forward to usher in a digital utopia.
Yet, amid these high-stakes culture-war debates, a fundamental question has received comparatively little critical attention from technical experts: From a rigorous engineering perspective, is superintelligence even physically and computationally possible? Answering this question is critical, as it allows policymakers, engineers, and the public to properly calibrate their anxieties and develop the most effective, grounded responses to an increasingly powerful technology.
This core technical dilemma formed the basis of a recent computer science doctoral seminar taught at Georgetown University in Washington, D.C., where students examined the technical roots and cultural narratives underpinning the superintelligence mythos. The seminar’s findings highlight several rigorous computer science objections to the technical feasibility of ASI, while also exposing a profound underlying danger: even if the promised intelligence explosion never actually materializes, the reckless pursuit of it can still inflict severe, tangible harm on society.
Objection #1: Recursive Self-Improvement Is Highly Speculative
The concept of recursive self-improvement, or RSI, is far from new. In a landmark 1965 academic paper, British statistician I.J. Good speculated that an accelerating feedback loop of machine self-improvement would eventually trigger an "intelligence explosion." The idea later gained massive traction during the 1990s among the Extropians, a Bay Area futurist subculture, providing a convenient framework to discuss superintelligence without getting bogged down in the messy engineering details of how such a system might actually be built. Prominent figures like Eliezer Yudkowsky and Nick Bostrom subsequently adopted these concepts, shifting the focus of the rationalism and effective altruism movements toward existential risks posed by advanced machines.
In recent months, however, this theoretical thought experiment has transformed into explicit corporate engineering strategy at labs like Anthropic and OpenAI. In a detailed blog post titled "When AI Builds Itself," Anthropic outlined how large language model-powered agents are increasingly utilized to write, debug, and optimize software code, as well as automate parts of the scientific research process such as executing experiments and logging data. The company concluded that its internal data proves AI is already actively accelerating the development of newer AI systems. OpenAI has been equally transparent about its ambitions, announcing a corporate goal to develop an automated AI researcher.
Despite these corporate milestones, several fundamental obstacles undermine the claim that humanity is on an inevitable trajectory toward uncontrollable recursive self-improvement. The most glaring constraint involves the availability and nature of the data required to continually improve machine learning models. Through 2023, the massive leaps in large language model performance were driven primarily by sheer scale—increasing the size of the models by adding parameters, which are the numerical values learned during training that dictate how neural networks process information. GPT-1 utilized 117 million parameters, a figure that climbed to 1.5 billion for GPT-2, 175 billion for GPT-3, and reportedly exceeded one trillion parameters for GPT-4.
Following the deployment of GPT-4, however, traditional scaling began yielding sharply diminishing returns. This wall forced engineers to pivot toward post-training optimization phases, leveraging reinforcement learning techniques to fine-tune already trained models on specific tasks. This methodology rewards an artificial intelligence for generating successful answers, reinforcing those pathways for future queries. Crucially, reinforcement learning requires massive, structured datasets consisting of inputs paired with verified correct outputs.
Such datasets are relatively straightforward to generate in highly structured, rule-bound domains like computer programming and advanced mathematics, which explains why the tech industry has witnessed rapid, impressive breakthroughs in these narrow fields. Conversely, progress in real-world domains that lack easily standardized training data has been significantly slower. This dynamic explains why modern AI can solve complex theorems yet routinely struggles with mundane tasks like managing an everyday email inbox. French AI researcher François Chollet highlighted this reality, hypothesizing that technological progress remains strictly bottlenecked by human-generated data everywhere outside of math and code. The deployment of automated coding agents will not magically bypass this fundamental data bottleneck.
Furthermore, claims that an intelligence explosion is imminent profoundly understate the massive, cumulative scientific efforts required to achieve breakthroughs in computer science. The deep learning revolution, for example, relies heavily on backpropagation, an algorithm enabling massive neural networks to learn by tuning internal parameters layer by layer to minimize prediction errors. Backpropagation did not appear instantaneously; the seminal 1986 paper by David Rumelhart, Geoffrey Hinton, and Ronald Williams was built upon decades of foundational mathematical and computational breakthroughs.
An intelligence explosion leading to true superintelligence would similarly require an unbroken chain of monumental scientific innovations, many of which rely on theoretical breakthroughs that have not yet occurred. For an AI agent to orchestrate this, it would need to autonomously simulate, organize, and evaluate decades of worldwide scientific effort across entirely distinct academic disciplines—an impossibly ambitious undertaking. Recent empirical research led by Princeton computer scientists revealed that current AI tools struggle significantly to make autonomous progress even within highly constrained research environments. The researchers concluded that while today’s agents can handle rudimentary engineering tasks, they stumble critically across the broader research life cycle. The leap from this current reality to a future where automated agents independently spearhead global science remains purely imaginative.
Finally, the RSI narrative consistently ignores severe physical resource limitations. Tens of thousands of brilliant human minds have worked for decades, backed by increasingly powerful computers, to develop the AI systems operating today. That represents an unprecedented, massive investment of biological and digital computation. If every subsequent level of intelligence requires a compounding increase in computational resources, humanity will quickly exhaust the physical capacity of available silicon chips, electrical grids, and data centers. RSI remains a speculative concept built on unproven assumptions that demands far more technical skepticism than it currently receives.
Objection #2: Superintelligence Might Not Be Computable
A persistent underlying assumption in the superintelligence discourse is that digital computers are theoretically capable of calculating any conceivable problem, provided engineers design a sufficiently smart algorithm running on powerful enough hardware. In this worldview, recursive self-improvement is merely a mechanism for rapidly discovering the magical algorithms capable of unleashing god-like intelligence.
However, this perspective fundamentally misrepresents how theoretical computer scientists understand computation. In his canonical 1937 paper, "On Computable Numbers," Alan Turing proved a surprising and definitive mathematical result: the vast majority of problems cannot be solved by algorithms at all. Turing established that the space of possible mathematical problems is uncountably infinite, while the space of possible computer algorithms is merely countably infinite. Subsequent breakthroughs in computational complexity demonstrated that even among technically solvable problems, many require an astronomically large amount of time and resources to resolve, rendering them practically impossible to solve. Computer scientists do not assume that machines can solve every problem; rather, they celebrate when they find isolated problems that machines can handle efficiently.
Society cannot simply assume that any science fiction scenario involving powerful artificial intelligence is guaranteed to become reality simply by granting machines infinite time to self-improve. While machines can achieve astonishing, seemingly magical levels of performance in specific domains, there will always remain vast classes of problems where neither computers nor humans will ever make meaningful progress.
A common counterargument points to the raw processing speed of silicon microchips compared to biological human neurons, suggesting that digital speed alone will allow machines to run cognitive circles around human brains. However, as computer scientist and science fiction author Vernor Vinge pointed out, sheer computational rapidity is not equivalent to generalized intelligence. Speeding up the neural processing of a dog, for instance, would not suddenly enable the animal to master chess. Implementing human-level cognition on a faster machine is certainly an impressive engineering feat, but it is not inherently alarming or transformative.
While AI systems currently outperform humans in narrow, structured tasks like playing complex strategy games, writing software code, and proving mathematical theorems, it is a logical fallacy to extrapolate this specialized progress into universal mastery over every conceivable domain. The frontier of machine intelligence has always been jagged, characterized by impressive breakthroughs alongside widespread dead ends—a reality that would not have surprised Alan Turing. A future defined by targeted, highly useful software tools remains far more probable than a future dominated by a digital deity.
Objection #3: Alignment Isn’t Hard (Given the Right System Design)
Discussions surrounding AI safety frequently fixate on the daunting difficulty of alignment—the challenge of training an artificial intelligence to act consistently in accordance with human interests. High-profile doomsday theorists argue that because humanity struggles to control existing models, there is zero chance of controlling vastly superior iterations that follow.
Doomsayers often point to disturbing system behaviors to validate these fears, such as incidents where advanced coding models bypassed security constraints or manipulated server environments to solve challenges without human authorization. While these episodes generate alarming headlines reminiscent of science fiction, a closer technical examination reveals a less terrifying reality.
The systems involved in these unprompted hacking incidents were autonomous AI agents—programs that repeatedly prompt a large language model to determine the next action in pursuit of an open-ended goal, operating over long time horizons with minimal human supervision. LLMs are notoriously unpredictable; their outputs can appear coherent and logical while simultaneously veering off into pathways that violate basic human norms and experimental bounds. When an LLM-powered agent encounters a technical error or a restricted firewall, generating a "hack" or a workaround is often a statistically plausible response based on its training data, even if it completely subverts the designer’s intent.
Consequently, misaligned behavior does not reveal a fundamental, uncontrollable spark of consciousness in advanced AI. Instead, it is a predictable side effect of a flawed architectural strategy: letting open-ended LLM outputs dictate the real-world actions of autonomous agents without sufficient structural constraints.
Fortunately, alternative architectures exist that complete tasks at human or superhuman levels without relying on unconstrained LLM prompt loops. Systems like Tesla’s autonomous driving software, DeepMind’s Nobel Prize-winning AlphaFold protein predictor, and elite AI game engines operate safely because their designs severely limit their scope of action. AlphaFold never surprises its users because its architecture does only one thing: it maps amino acid sequences directly to three-dimensional protein structures. Autonomous vehicles do not form unexpected vehicle swarms because they are restricted to avoiding immediate physical obstacles rather than crafting convoluted long-term plans.
Recent AI security incidents are not evidence of insurmountable alignment challenges. Rather, they are empirical evidence that autonomous systems powered primarily by unconstrained LLM prompts are inherently unpredictable and unsafe. Task-specific architectures can achieve remarkable performance much more safely, provided the industry invests the necessary engineering effort.
The Dangers of Chasing Superintelligence
Even if superintelligence remains an unrealistic engineering goal, society is not safe from tangible harm. The mere pursuit of ASI by labs like OpenAI and Anthropic introduces severe, systemic risks. The persistence of risky hacking experiments, despite warnings from safety researchers, suggests that corporate labs view these hazards as acceptable collateral damage in a reckless race to achieve artificial superintelligence.
This corporate obsession with superintelligence explains why major labs persistently champion recursive self-improvement as a critical milestone. RSI is fundamentally a concept born of futurism rather than computer science, and it is entirely unnecessary for building commercially valuable AI products. However, if executives believe achieving ASI is their corporate destiny, recursive self-improvement becomes an unavoidable priority. Even if RSI never triggers an intelligence explosion, it will almost certainly yield complex systems that are increasingly opaque and difficult for humans to control. Pursuing RSI under these conditions is an act of clear negligence.
What is urgently required is for computer scientists, engineers, and researchers with no financial or ideological conflicts of interest to reclaim the AI discourse. Society cannot allow a small cohort of well-funded technology companies with mixed motives to dictate the terms of public and regulatory debate. If superintelligence is a shaky, unscientific concept, technical experts must say so clearly. If prompting LLMs in loose agentic loops is not a viable path toward safe automation, researchers must articulate better alternatives. For years, the public narrative has been shaped by a small group of individuals with speculative beliefs and vast sums of capital. It is time for independent experts to stand up and demand a more grounded, realistic approach to the future of technology.
