The Wrong Panic: Washington Fears Superintelligence While Automation Bias Spreads

Foreign Affairs Forum | Dr. Antonio Bhardwaj (Dr. 🆎)| November 3rd, 2026

Executive Summary

Washington's debate about artificial intelligence has been captured by a single narrative: that a superintelligent system, built within years by a handful of frontier laboratories, could slip human control and extinguish the species.

Insiders at those laboratories have placed the odds of such an outcome at greater than one in ten and have suggested that it could materialize before the end of the decade. Activists with a background in nuclear disarmament have reinforced the alarm by pointing out that humanity has built civilization-ending technologies before.

The doomsayers may yet prove correct, but their scenario depends on a long chain of assumptions about capability, autonomy, deception, and the failure of every safeguard, each of which is uncertain and all of which must hold simultaneously.

The consequence of this fixation is that policymakers are neglecting a danger that is neither speculative nor distant.

Automation bias, the tendency of human beings to over-rely on machine outputs relative to their actual accuracy, to off-load judgment to the system, and sometimes to defer to it even in the face of contradictory evidence, is already operating in hospitals, courtrooms, welfare offices, financial institutions, newsrooms, and military command posts. It is mundane, widespread, and measurable, and it scales with every deployment. It also compounds the catastrophic risks that dominate the Washington conversation, because a society that cannot calibrate its trust in today's imperfect systems will not be able to supervise tomorrow's powerful ones.

FAF analysis traces the intellectual and policy history of the American AI debate, describes the present balance between doom-driven and harm-driven approaches, and reviews recent developments in legislation, executive action, litigation, and warfare. It then undertakes a cause-and-effect analysis of how automation bias arises and spreads, and offers concrete steps for institutions that wish to govern it.

Dr. 🆎, who has spent a career examining human-centered superintelligence in geopolitical strategy, argues that the central question of the coming decade is not whether machines will outthink us but whether human beings will remain capable of thinking alongside them. His warning, developed throughout this essay, is that the most dangerous system is not the one that deceives its overseers but the one whose overseers have ceased to check.

Introduction

Every technological revolution produces its own apocalyptic literature.

The railways were said to endanger the sanity of passengers, electricity to threaten the moral order, and nuclear fission, with considerably more justification, to place the survival of civilization in the hands of a few governments.

Artificial intelligence has inherited this tradition and added to it a distinctive feature: the possibility that the technology itself might acquire interests of its own. The idea has moved from the margins of philosophy seminars to the center of American policy, endorsed by laboratory insiders, amplified by activists, and invoked by legislators who propose to pause or ban the development of advanced systems.

The seriousness of these concerns should not be dismissed.

Dr. Antonio Bhardwaj (Dr. 🆎), a polymath whose expertise spans superintelligence, geopolitical strategy, artificial intelligence in warfare, and the governance of biological risk, has long argued that low-probability, high-consequence scenarios deserve rigorous attention, and he has written extensively on the potential for advanced models to lower the barriers to biological weapons development. His critique is therefore not that the catastrophic scenarios are fantasies but that they are crowding out a more tractable and more immediate problem. In his words, a nation that spends all its attention on the asteroid while the roof is collapsing will lose the house before it ever meets the rock.

The argument of this article is accordingly one of proportion rather than denial. It holds that the policy landscape is poorly calibrated to the distribution of actual harms, and that the most pressing risk is not a machine that escapes human supervision but a human supervisor who has quietly abandoned the task.

The article proceeds by examining the history of automation bias and the evolution of the American debate, then the current state of play, followed by the key developments of recent months and the latest facts and concerns. It then analyzes the causal mechanisms at work, sets out practical steps for governments, firms, and professions, and concludes with a reflection on what a mature approach to AI governance would look like.

History and Current Status

Automation bias is not a new phenomenon.

The aviation industry documented it in the late 20th century, when pilots who relied on autopilots and flight management systems failed to notice errors that were obvious from the instruments, or followed automated advice into situations that a vigilant human would have avoided.

Researchers described a cluster of related failures: errors of commission, in which operators follow incorrect recommendations, and errors of omission, in which they fail to act because the system does not prompt them.

The phenomenon was later observed in medicine, where clinicians presented with computer-aided diagnoses sometimes accepted incorrect ones, and in nuclear and industrial control rooms, where operators learned to trust displays over their own judgment.

What distinguishes the present era is the combination of scale and fluency.

Earlier systems were narrow, rule-based, and transparent about their limits. Contemporary generative models are general, probabilistic, and conversational, and they present their outputs in a confident, polished register that invites trust. They are also embedded in everyday workflows by default, so that millions of professionals and citizens interact with them daily. The result is that the conditions that produce automation bias, namely cognitive load, time pressure, institutional incentives to move quickly, and an interface that conveys authority, are now present across virtually every domain of knowledge work.

The most instructive episodes in this history are those in which the machine was not malfunctioning at all.

Investigators of aviation accidents have repeatedly found that systems performed as designed while crews, accustomed to trusting them, failed to monitor what they were doing, and that the resulting catastrophes were failures of the partnership rather than of the technology. The lesson drawn by the safety community was that reliability breeds complacency, and that the more dependable an automated system becomes, the more carefully its human supervisors must be trained, tested, and supported.

That lesson has yet to be absorbed by the institutions now adopting generative models, which are often introduced with the promise that they will reduce the burden on human staff rather than with a plan to preserve the quality of their oversight.

The American policy debate has evolved along a different track.

After the release of powerful chatbots in 2022 and 2023, public attention turned to existential risk, and a series of open letters, summits, and testimony by laboratory leaders placed the possibility of human extinction on the national agenda.

The Biden administration emphasized safety testing and voluntary commitments, while the Trump administration, after returning to office in 2025, rescinded much of that framework and prioritized competitiveness against China, while also seeking to limit the reach of state regulation.

In the months since, the debate has acquired a quasi-religious intensity, with some advocates demanding a moratorium on frontier development and others dismissing all regulation as a gift to Beijing.

The present state of play is one of legislative paralysis and executive improvisation.

On Sep 29th, the President and the leaders of major laboratories, including Google, Anthropic, Meta, OpenAI, and Nvidia, signed an accord committing the companies to internal controls, oversight teams, and independent external auditors, an arrangement that the White House described as morally binding but that has no statutory force.

The administration has also announced an AI Force and a forthcoming AI czar, directed agencies to adopt the term superintelligence, and agreed with China to launch a dialogue on superintelligence and a channel for reporting AI incidents. Vice President Vance has rejected the idea of a dedicated regulator of the kind that oversees aviation or pharmaceuticals, arguing that existing authorities suffice.

Congress, meanwhile, has produced much heat and little law.

Proposals range from an emergency shutdown mechanism for advanced systems, which was blocked in the Senate on Sep 16th, to a bill that would ban the development of artificial superintelligence until a new federal department sets safety rules, introduced by Sen. Bernie Sanders and Rep. Greg Casar. A bipartisan risk-management bill sponsored by Sens.

Mark Warner, Brian Schatz, and Andy Kim met an objection from Sen. Ted Cruz on Sep 29th, and a frontier bill reportedly negotiated by Cruz, Majority Leader John Thune, and Sen. Amy Klobuchar may preempt state safety laws. A coalition of twenty-six state attorneys general has urged Congress to legislate while preserving state authority, and California has enacted new certification and auditing regimes for AI evaluators.

Key Developments

The first development is the maturation of the doom narrative into a political program.

What began as a debate among researchers has become a legislative agenda, with bills to pause development, create emergency buttons, and establish new departments. The advocates are sincere and in some respects correct: frontier laboratories are building systems whose capabilities exceed their creators' ability to predict, and the incentives of commercial competition militate against caution. Yet the doom framing has distorted the policy conversation by directing attention to scenarios that are hard to measure, hard to legislate against, and hard to falsify.

The second development is the accumulation of evidence on how humans actually behave when working with AI.

A study of West Point cadets, comparing them with members of the general public in a target identification task, found that the cadets showed better-calibrated trust in algorithmic advice and were less prone to cognitive distortions than civilians, suggesting that training and institutional culture can shape the relationship between human and machine. A separate experiment involving more than two thousand Israeli military personnel using a replica of a targeting support system found evidence of algorithmic aversion in high-stakes scenarios with significant collateral damage, and found that explainability features reduced that aversion and encouraged more careful evaluation of recommendations. These findings complicate the simple story that humans always defer to machines. They show instead that trust is context-dependent and malleable, which is precisely why it can be managed.

The third development is the use of AI-enabled decision support in armed conflict.

Reporting on the war in Gaza described a system that generated tens of thousands of candidate targets and, according to some accounts, was approved by human operators in seconds, raising the specter of oversight reduced to a rubber stamp. Whatever the exact facts, the episode illustrates the structural problem: when a machine produces recommendations at a speed and volume that exceed human capacity to review them, nominal human control can become a fiction. The current war involving the United States and Iran, in which sensors, drones, and algorithmic analysis play an expanding role, makes the lesson urgent.

Dr. 🆎 has argued that the true measure of human control is not whether a person signs the order but whether that person has the time, information, and authority to refuse it.

The fourth development is the entry of AI into the institutions of ordinary governance.

Courts have sanctioned lawyers who submitted briefs containing fabricated citations generated by chatbots. Hospitals are deploying ambient note-taking and diagnostic assistants. Welfare agencies use automated systems to flag fraud and allocate benefits, with documented cases around the world of wrongful denials and accusations. Employers rely on algorithmic screening and, in the language of one California bill, risk becoming robo-bosses. In each case, the danger is not that the machine plots against its users but that its users, pressed for time and reassured by its fluency, stop exercising independent judgment.

The fifth development is the emergence of litigation and regulatory enforcement as a de facto governance mechanism.

The Federal Trade Commission has opened an industry-wide investigation into consumer harms from AI laboratories, a state attorney general has asked a court to bar a leading company from developing new models without independent safeguards, and the courts have begun to address copyright, competition, and national security questions. A divided federal appeals court allowed the Pentagon to designate a leading laboratory as a supply chain risk, a ruling that illustrates how the government's own procurement and security tools are shaping the market. These developments suggest that the United States is governing AI by accident, through a patchwork of cases and agency actions, rather than by design.

A sixth development is the growing recognition among laboratories and researchers themselves that human-machine interaction deserves attention.

Several leading companies have published work on sycophancy, the tendency of models to flatter and agree with users, and on calibration, the alignment between a model's expressed confidence and its actual accuracy. These are encouraging signs, but they also reveal the limits of a purely technical response. A model that is honest about its uncertainty helps only if the human reader notices, understands, and acts on that signal, and there is little evidence that busy professionals do so consistently.

The design of the interface, the structure of the workflow, and the culture of the organization determine whether the signal is heeded, and none of these is addressed by laboratory commitments alone.

Latest Facts and Concerns

The latest facts reveal an environment of rapid deployment and thin oversight.

The accord signed on Sep 29th relies on voluntary commitments, and the administration has rejected a binding regulator. Congress has not passed a comprehensive statute. States are acting independently, and the prospect of federal preemption has provoked opposition from civil society groups and from a bipartisan coalition of attorneys general. The result is a regulatory landscape in which firms face inconsistent obligations and citizens face inconsistent protections, while the technology itself advances month by month.

Three concerns deserve emphasis.

The first is that automation bias is difficult to detect because it manifests as the absence of an action rather than the presence of an error.

A clinician who accepts an incorrect recommendation, a judge who adopts a flawed risk score, or an analyst who forwards a machine-generated assessment without checking its sources leaves little trace of the lapse, which makes it hard to measure and easy to ignore. Institutions that do not look for the phenomenon will not find it, and those that do not find it will assume that it does not exist.

The second concern is that the incentives of deployment push toward greater reliance rather than less.

Organizations adopt AI in order to save time and money, and the savings accrue only if human review is minimized. A hospital that requires a physician to verify every machine-generated note in detail forfeits the productivity gain, and a government agency that insists on a full human reassessment of every automated decision loses the efficiency that justified the purchase. The temptation to reduce oversight is therefore structural, and it will intensify as models improve, because better models produce fewer visible errors and thus lull their overseers into complacency.

The third concern is that automation bias interacts dangerously with the very scenarios that the doom narrative emphasizes.

If a society habitually defers to machine judgment, then the moment at which a genuinely dangerous system emerges is precisely the moment at which humans are least equipped to notice.

Dr. 🆎 has argued that this is the neglected link between the mundane and the catastrophic: the capacity for critical oversight is a muscle, and it atrophies when unused. A civilization that has outsourced its routine cognition to machines will find it difficult to summon the skepticism needed to question them when it matters most.

Concerns also attach to the security landscape.

Dr. 🆎 has warned that advanced models can lower the barriers to dangerous biological and cyber capabilities, and that the combination of capable tools and uncritical users is especially hazardous. A laboratory technician who accepts a model's suggested protocol without understanding it, a cybersecurity team that trusts an automated alert without verification, or a commander who acts on a machine-generated assessment without challenge may each cause harm without malicious intent.

Meanwhile, deliberate misuse by hostile stakeholders is amplified when defenders are over-reliant on automated detection that can be spoofed or manipulated. These are not science-fiction scenarios but extensions of failures already observed.

Cause-and-Effect Analysis

The causal chain that produces automation bias begins with the cognitive architecture of human beings.

People economize on mental effort, and when a source appears reliable, they tend to treat its output as a substitute for their own evaluation. This tendency is reinforced by social proof, since if colleagues and institutions use the system, individuals assume that it has been vetted. It is further reinforced by the interface: a system that speaks in complete, confident sentences and offers rapid answers triggers the same heuristics that people apply to experts.

The second link is organizational. Institutions design workflows around the assumption that the machine is usually right, and they measure performance by throughput.

Employees who pause to verify outputs risk being slower than their peers, and those who challenge the system risk being blamed if they are wrong and unrewarded if they are right. In such an environment, deference becomes the rational individual strategy, even if it is collectively dangerous. The effect is to convert a psychological tendency into an institutional norm.

The third link is informational.

Because most deployments do not systematically record the cases in which humans overrode or accepted the system, and do not compare outcomes against ground truth, organizations lack the data needed to calibrate trust. They cannot say how often the model is wrong, in which contexts, or how often reviewers catch its errors. This opacity allows overconfidence to persist, and it allows vendors to market their products on the basis of average accuracy while ignoring the distribution of errors that matters for high-stakes decisions.

The fourth link is political.

The doom narrative has an economic and rhetorical advantage: it is dramatic, it attracts attention and funding, and it invites sweeping solutions. Mundane risks, by contrast, are diffuse and technical, and their victims are often individuals who lack political voice, such as welfare claimants or defendants. Legislators respond to what is salient, and salience favors the catastrophic. The result is a policy agenda that is rich in proposals for pauses, buttons, and new departments and poor in requirements for audit trails, error reporting, and human review standards.

The fifth link connects the domestic to the international.

The United States is competing with China for leadership in artificial intelligence, and both governments fear that excessive caution will cede advantage to the other. This competition encourages rapid deployment, including in military and intelligence settings, where the pressure to shorten decision cycles is intense. The dialogue on superintelligence agreed by Washington and Beijing is a welcome step, but its focus on catastrophic risk may overlook the more immediate danger that both militaries are deploying systems whose recommendations their operators are poorly equipped to evaluate.

The sixth link is technological.

As models become more capable, they will be more persuasive and more accurate on average, and therefore more trusted. But average accuracy is not the same as reliability in the tail, and the cases in which a highly capable system errs may be those in which the error is most convincing and most consequential. This is the paradox of progress: the better the machine, the greater the danger of unexamined reliance. A policy that assumes that improvement in capability reduces the need for oversight has the logic backwards.

The seventh link, and the one most easily overlooked, is cultural.

Societies that celebrate technological progress tend to treat skepticism toward machines as obstruction, and employees who question automated outputs are sometimes characterized as resistant to change. This cultural framing penalizes exactly the behavior that safety requires. A healthier norm would treat verification as a mark of professionalism rather than of timidity, in the way that surgeons confirm the site of an operation and pilots run through checklists regardless of experience. Cultural change of this kind is slow, but it can be accelerated by leadership, by incentives, and by visible examples of verification preventing harm.

Dr. 🆎 synthesizes these links in a single observation. A human-centered approach to superintelligence, he argues, begins from the premise that the relevant system is not the model alone but the human-machine team, and that safety is a property of the team. A model that is excellent but poorly integrated with its human partners can be more dangerous than a modest one that is well integrated. This perspective shifts the policy question from how to restrain the machine to how to design the relationship, and it suggests that the most valuable safeguards are those that preserve and strengthen human judgment rather than those that merely constrain model behavior.

Future Steps

The first step is to make automation bias a recognized category of risk in federal policy.

The administration and Congress should direct agencies to assess, in every high-stakes deployment, how human reviewers interact with machine outputs, and to report the rates at which recommendations are accepted, modified, and overridden. Such measurement is the precondition of governance, since a risk that is not measured cannot be managed. The National Institute of Standards and Technology is well positioned to develop standards for this purpose, building on its existing risk management framework.

A second step is to require meaningful human review in consequential decisions.

Statutes and regulations should specify that when an automated system informs a decision affecting liberty, health, livelihood, or life, the human reviewer must have access to the underlying evidence, sufficient time to evaluate it, and the authority to reject the recommendation without penalty. Nominal human-in-the-loop requirements that can be satisfied by a signature are insufficient. The evidence from experiments with explainability features suggests that interface design can encourage more careful evaluation, and regulators should encourage designs that prompt reviewers to engage rather than to acquiesce.

A third step is to invest in training and professional norms.

The finding that cadets exhibit better-calibrated trust than civilians indicates that education can shape behavior. Medical schools, law schools, military academies, and public administration programs should teach students how and when to trust machine outputs, how to test them, and how to recognize signs of over-reliance. Professional bodies should adopt standards of practice for the use of AI, including documentation requirements and sanctions for negligent reliance.

A fourth step is to align incentives.

Organizations should reward employees who identify errors in machine outputs, and should evaluate performance on the basis of outcomes rather than speed alone. Procurement rules should require vendors to disclose the error profiles of their systems, including performance across subgroups and in edge cases, and to support independent evaluation. The auditing regimes beginning to emerge in California and in the voluntary accord offer a foundation, but they must be extended to cover the human side of the equation as well as the technical one.

A fifth step concerns the military and intelligence communities.

Defense and intelligence agencies should adopt doctrine that defines meaningful human control in terms of time, information, and authority, and should test their operators against the failure modes of automation bias in realistic exercises. Allies should share lessons, and the United States and China should extend their incident-reporting channel to cover failures of human oversight as well as accidents involving autonomous systems. The war in the Gulf has demonstrated that the pace of modern operations can overwhelm traditional review processes, and doctrine must adapt before the next conflict.

A sixth step is to restore balance in the national debate.

Catastrophic risks deserve serious investment, including in evaluation of dangerous capabilities, biosecurity safeguards, and international verification, and Dr. 🆎 has urged that these efforts continue. But they should not be pursued at the expense of the nearer-term work. A mature policy portfolio would allocate resources across the spectrum of risks according to their likelihood and severity, and would recognize that strengthening human oversight today is also the best preparation for managing more powerful systems tomorrow.

Finally, the argument for proportion has an international dimension.

Other governments are watching how Washington frames the problem, and many are drafting their own rules. The European Union has opted for risk-based regulation that includes requirements for human oversight of high-risk systems, though critics note that oversight mandates are only as good as the competence and independence of the humans assigned to exercise them. If the United States defines AI safety exclusively in terms of preventing a rogue superintelligence, it will cede the more practical conversation about human oversight to others and will find itself importing standards written elsewhere.

A confident American approach would instead lead in developing the science and practice of calibrated human-machine collaboration, exporting not only the technology but the institutions that make it trustworthy.

Looking toward 2030 and 2036, the nations that build robust institutions of human-machine collaboration will be better placed to benefit from the technology and to withstand its failures.

Conclusion

The debate in Washington about artificial intelligence has the character of a drama in which the audience is transfixed by a distant and terrible villain while a quieter and more familiar one is already at work.

The villain on the horizon may be real, and it deserves vigilance. But the villain at work is the tendency of human beings to hand over their judgment to machines that are fluent, fast, and frequently wrong, and it is operating now, in hospitals, courts, agencies, and command centers, with consequences that accumulate unnoticed.

The remedy is not to abandon the effort to prevent catastrophe but to broaden it.

A society that learns to calibrate its trust in imperfect systems, to measure the performance of human-machine teams, to preserve the capacity for critical review, and to reward those who exercise it will have built the foundations of any serious defense against more dangerous machines. A society that does not will find that its safeguards were hollow when they were needed most.

Dr. 🆎 offers a final formulation: the ultimate safeguard against superintelligence is not a switch or a ban but a population, and a set of institutions, that remain intellectually sovereign. If Washington can redirect some of the energy it now devotes to imagining the end of the world toward the humbler task of keeping humans genuinely in command of the decisions that shape their lives, it will have done more for safety than any dramatic gesture could achieve. The forest, after all, is made of the trees.