When machines go rogue: What the summer of AI agent chaos teaches us about engineering trust

Stephen West

The summer of AI agent chaos shows why government departments must engineer trust into their architecture to protect against autonomous AI risks from both outside and within.

When I delivered a talk at a recent cybersecurity community conference, my goal was to amplify the alarm on a fundamental shift in computing: the transition of agentic AI from a directed tool to an autonomous actor. My focus was on complex cyberattacks, whilst emphasising that the controls needed to defend against them are fundamentally the same as those needed to keep internal AI agents safe and within the boundaries of delegated authority.

We are no longer just dealing with systems that require human prompting. Agentic AI systems are now capable of real-time reasoning, decision-making, and multi-step execution directly into corporate networks, software supply chains, and government workflows. These deployments are made with both good and malicious intent, with the risks remarkably similar.

In my talk at the CyNam event, I argued that standard security controls, even when implemented correctly, would fail against machine-speed, autonomous threat vectors.

Little did I know how rapidly the real world would validate that thesis.

The reality hits: OpenAI & Hugging Face, Anthropic, AISI, and Meta

In just the past few weeks, a wave of incidents involving AI agents “breaking out” has demonstrated why trusting autonomous systems without strict, thoughtful systems architecture and security controls is a recipe for disaster. 

In fact, I have tried to write this blog several times now. Each time I sit down, open my laptop and start typing, another major AI agent security incident hits the news!

These incidents illustrate what I said at CyNam: when agents inherit or acquire over-privileged identities and act on soft system prompts, static rule engines can become ineffective. It should be borne in mind, however, that very many such tests are being carried out. The fact that four such incidents (that I’ve seen) have made the news, along with a number of reported autonomous cyberattacks, is of concern, but not necessarily the sign of an impending cataclysm.

It should also be noted that the design of the tests, and so the conditions for these incidents, may well have been deliberate choices. AISI states in its report that whilst its evaluation design choices may have enabled the behaviour, the agent still demonstrated novel and potentially deceptive behaviours not anticipated.

It is, however, a serious warning sign that needs to be acted upon with immediacy.

How we engineer trust in government AI agents: The Zero Trust imperative 

The emerging risk of rogue autonomous AI agents requires a systemic approach in government, as it arises from the complex interaction of diverse stakeholders, evolving cyber threats, market forces, regulation, government policy, rapid AI advancement, and legacy technology. As I outlined in my talk, an important activity for all organisations is to build engineering trust.

  1. Zero Trust Architecture (ZTA): Help defend against autonomous AI threats by continuously verifying identities, enforcing least-privilege access, segmenting networks, and monitoring behaviour. These controls limit an AI agent’s ability to move laterally, access unauthorised resources, or exfiltrate data, whilst enabling rapid detection, containment, and response to anomalous or malicious activity. 
  2. Keyless authentication: Implement ZTA patterns using stronger hardware and contextual-based authentication. Just-in-time access, especially for privileged access, uses short-lived, ephemeral tokens so stolen credentials are useless out of context.
  3. Validate execution threads: Static rule engines can be found wanting in these scenarios. Instead, trust must be continuously validated at execution time through behavioural profiling and execution-thread verification, enabling real-time detection and containment of anomalous behaviours from both machines and humans.
  4. Hard egress boundaries: Establish hard service perimeters around key system components, such as databases and API gateways, to tightly control access and exfiltration through intelligent, behaviour and context-aware rules. And of course network segregation (to limit access to sensitive environments, e.g. jumping from test to live) and segmentation (to limit lateral movement and blast radius) remain  an important foundation for IT security. 
  5. Humans as system governors: Human-in-the-loop validation focussed solely on task-checking is inadequate at machine speed. Automated fastpaths are insufficient for managing autonomous AI risk. Effective control requires robust guardrails combined with targeted human intervention at critical  decision points. It means creating ‘cognitive speed bumps’ to review important decisions before the AI can effect high-impact actions.

The era of treating AI as just a next-gen software application is officially over. If recent headlines prove anything, it’s that we must treat autonomous agents as non-human actors. They require the highest standards of Zero Trust governance, continuous monitoring, and hard containment with human intervention crucial for decisions and actions of consequence.

As AISI wrote within the conclusion of their report:

“Incidents of this kind reflect the speed at which AI is developing. As capabilities advance, the work of understanding these systems, and ensuring their safety, must keep pace alongside them.”

Now, if you’ll excuse me, I’m going to hit “Publish” on this blog before another incident lands in my news feed!

If you’d like a recording of my CyNam presentation or discuss the implications of deploying a secure AI agent in government workflows, feel free to get in touch or connect with me on LinkedIn. 

To explore how Zaizi builds safe, secure, and compliant digital services for government and critical infrastructure, download our whitepaper: “Unlocking government efficiency: An AI readiness roadmap for decision makers.”

Related content

Thanks for joining us! We’ll keep you informed with regular updates.

[gravityform id=”16″ title=”true” ajax=”true”]