Op-ed: What happens when AI can replicate itself autonomously and we can no longer control it?
Experiments by Anthropic on Autonomous Replication and Adaptation and the risks beyond the scenarios tested.
Anthropic, a leading AI company best known for developing Claude, recently published research on Autonomous Replication and Adaptation (ARA), exploring whether AI systems can replicate and adapt without human intervention.
The experiments investigate whether relatively small AI models could create working copies of themselves and spread autonomously across Internet of Things (IoT) devices. These include everyday internet-connected devices such as smart TVs, security cameras and household appliances.
Anthropic, however, places a strong emphasis on extreme AI risks, and this perspective may influence how its findings are presented. It is therefore important to distinguish between what the experiments actually demonstrate and the scenarios they anticipate.
Notably, the experiments use open-weight AI models, whose model weights can be downloaded and run independently, without ongoing access to the infrastructure of the company that developed them. By contrast, Anthropic primarily makes its Claude models available through services it controls.
This distinction also raises questions about the commercial interests behind the debate over the safety of open AI models. Research highlighting the risks associated with these models can shape public perceptions and regulatory decisions in an industry where developers of proprietary models have their own economic interests. This does not mean Anthropic is deliberately trying to discredit open models, but the commercial context deserves consideration. Moreover, the risks associated with AI autonomy are not limited to open-weight models.
There are several important technical limitations.
First, even a “small” AI model is enormous compared with conventional malware. It may require gigabytes of storage, substantial memory and significant computing resources.
Second, running these models on IoT devices is extremely slow. In a separate demonstration, Anthropic ran a model distributed across 15 Android TV devices, achieving a speed of just 0.39 tokens per second.
Third, conventional malware already spreads across vulnerable devices without carrying an AI model. Adding one requires additional resources, increases network traffic and potentially creates more opportunities for detection.
Anthropic acknowledges these limitations and does not claim to have demonstrated a small AI system capable of fully autonomous self-replication in the real world. But the implications of this technology deserve to be examined beyond the scenario of an AI model moving from one smart TV to another.
Stuxnet offers a useful historical comparison.
By the autumn of 2010, Stuxnet had infected approximately 100,000 computers, despite having the highly specific objective of sabotaging uranium-enrichment centrifuges at Natanz, Iran. Its modus operandi involved widespread propagation, persistence, concealment and the activation of destructive behaviour only under specific conditions. Stuxnet was automated, but its objective and activation conditions had been predetermined.
An AI system capable of autonomously creating working copies of itself could introduce a fundamentally different element through its ability to assess unfamiliar environments, adapt its decisions to changing circumstances and continue pursuing an objective without constant human intervention.
Such autonomy could make an operation far more adaptable, but also much harder for its own operators to predict, redirect or stop. The system would not even need to develop objectives of its own. It could continue pursuing its original purpose, but through decisions and behaviours that its operators could no longer control.
And if such capabilities become technically feasible, it is reasonable to assume that some states will be tempted to use them to gain a strategic advantage.
This brings us to what I consider the central concern. The very autonomy that could make a strategic operation more effective could also drastically reduce human control over it.
Anthropic already discusses the risks of autonomous propagation, but the implications for international security, and for maintaining control over systems capable of spreading, adapting and persisting without human supervision, remain poorly understood.
The consequences of such a scenario could extend far beyond the hacking of smart devices in our homes, concerning as that risk is in itself.
For speaking engagements or discussions on this topic or related subjects, you can contact the author at cristian@pauseai.info.
The views expressed in this article are those of the author and do not necessarily reflect the official position of PauseAI Romania.