Op-ed: AI-Enabled Terrorism and the Gray Swan Risk: New Research and the Challenge of Early Warning
New research examines how AI could contribute to a major terrorist attack and the challenge of preparing for a foreseeable but uncertain threat.
The gap between warning and prevention.
One of the central challenges in AI safety is the gap between recognizing a potential risk and taking effective measures to prevent it. As we have AI capabilities advancing, evidence of emerging dangers may remain incomplete, making it difficult to determine when preventive action is justified and what form it should take.
Naturally, when the likelihood of an event is uncertain but its potential consequences could be catastrophic this is at least challenging. Waiting for conclusive evidence may mean acting too late, while taking preventive measures requires careful consideration of risks that may never materialize.
We have a new paper now published by the Belfer Center for Science and International Affairs at Harvard Kennedy School that examines one particularly concerning dimension of this problem through the possibility of a major AI-enabled terrorist attack.
Anticipating a gray swan.
Not every catastrophic event is entirely unforeseen. Some risks can be identified well in advance, even when their likelihood, timing and precise circumstances remain uncertain. Such events are sometimes described as gray swans (in contrast to the more familiar concept of a black swan).
The researchers, including Stephen Casper, approach the possibility of a major AI enabled terrorist attack through this lens. They describe their hypothetical scenarios as gray swan events because, although considered unlikely, they could have exceptionally serious consequences and remain within the realm of predictability.
The research examines how AI could help enable a major cyberattack or an attack involving biological or chemical weapons. Rather than predicting when such an event might occur, the researchers use a method known as a pre-mortem. They assume, hypothetically, that an attack has already happened and work backwards to identify which decisions and failures could have made it possible.
They present nine testable predictions and two illustrative scenarios, focusing particularly on choices made by AI developers, distributors and governments. A central argument is that the risk depends not only on what AI systems become capable of doing, but also on how they are released and controlled.
Models accessible through commercial services may be vulnerable when safeguards and monitoring are inadequate. Models that can be downloaded and modified independently present different challenges, since their safety restrictions may be removed and their use becomes difficult to monitor.
The gray swan perspective is important because it shifts attention away from trying to predict a disaster with certainty and towards understanding which conditions could make it possible. The question is not whether such an attack is inevitable, but whether foreseeable vulnerabilities can be addressed before they contribute to a real-world event.
From recognizing risks to acting on them.
The challenge posed by a gray swan is not simply recognizing the warning signs, but establishing what should happen when those signs begin to appear.
In practice, this requires more than publishing risk assessments. Institutions need to identify which developments warrant closer scrutiny, what level of evidence should trigger intervention, and who is responsible for making that decision.
The study recommends independent safety evaluations, clearer thresholds for deployment decisions, incident reporting and structured information sharing between AI developers and governments. These are not merely regulatory questions but rather essential components of an early-warning system capable of connecting technical findings with preventive action.
For the European Union, some of the necessary foundations already exist. The EU AI Act establishes obligations for providers of general-purpose AI models classified as posing systemic risk, including risk assessments, adversarial testing and serious-incident reporting. The important question is how effectively such mechanisms can translate evidence of dangerous capabilities into timely decisions.
EU institutions and national authorities should therefore establish clear procedures for assessing and escalating credible warnings. These should include independent evaluations before the release of highly capable models, defined criteria for requiring additional safeguards, and proportionate restrictions where serious risks cannot be adequately mitigated.
Equally important is the flow of information between institutions. A warning identified by one AI developer may appear isolated, while similar activity detected across several providers could indicate a developing threat. Secure information-sharing arrangements should allow relevant findings to reach the EU AI Office, national competent authorities, cybersecurity agencies and, where appropriate, law enforcement.
Such arrangements also need clear responsibilities.
Who evaluates the credibility of a warning? Who decides whether further investigation is necessary? And which authority can intervene if the evidence becomes sufficiently concerning?
These questions should be resolved before a crisis, not during one. Joint preparedness exercises involving AI developers and relevant public authorities could help test whether warning indicators are recognized, information reaches the right institutions and decisions can be made within a useful timeframe.
None of this requires treating hypothetical threats as certainties, preventive measures must remain proportionate to the available evidence and account for the costs of unnecessary intervention. But the possibility of a gray swan presents a particular challenge for AI governance. The value of early warning ultimately depends on whether institutions have the capacity, procedures and authority to act while prevention is still possible.
For speaking engagements or discussions on this topic or related subjects, you can contact the author at cristian@pauseai.info.
The views expressed in this article are those of the author and do not necessarily reflect the official position of PauseAI Romania.