AI agents are finding ways around their controls. How do you test yours?

Most discussions about AI agents focus on what they’ve been told to do. I’m more interested in what they do when they find a way around the controls they’re supposed to operate within.

Earlier this month, I was quoted in an SC Magazine article about the OpenAI and Hugging Face security incident. There was a lot of talk at the time about an AI agent having “gone rogue”, which wasn’t really what had happened. 

As I said in the article: 

“From what we know so far, the agent was trying to achieve a specific objective and found ways to bypass the controls that were supposed to contain it.” 

Since then, we’ve learned more about what happened and there have been similar disclosures involving Anthropic and Meta. The circumstances aren’t exactly the same, but they all raise questions about how AI agents are being tested and how much confidence we can place in the environments and controls around them. 

Then this week there was a much more everyday example in Australia. 

A man had been using an AI agent to help with everyday tasks and asked it to book him into a popular Pilates class. The agent found a vulnerability in the gym’s booking system that allowed it to make bookings much further in advance than the rules allowed. 

Andrew was also fourth on the waiting list for another class and asked whether the agent could move him to the top. It found there were no proper authorisation checks and while testing what the system would allow, removed another gym-goer who was ahead of him. Andrew hadn’t asked it to remove anyone. 

There’s also a prompt engineering lesson here. Security controls and permissions are important, but so is the way the task is presented to the agent. If an agent is told to achieve an outcome, it may focus on that outcome above all else. A more cautious instruction might be to identify possible options and report them back rather than acting automatically. As agents become more capable, prompt engineering is becoming an important part of the guardrails around them, helping to define not just what an agent should achieve, but how it should behave while trying to achieve it. 

It’s obviously very different to an agent accessing another organisation’s systems, but I think it’s a useful example. The agent was given something to achieve, found a weakness and used it in a way its user hadn’t expected. 

As agents start doing more of these fairly ordinary tasks inside businesses, we need to understand what happens when they come across permissions, credentials or vulnerabilities that let them do more than we intended. 

If you’re already using agents, I’d be asking a fairly simple question. Have you tested what happens when one doesn’t behave in the way you expect? 

We’ve been doing this with other technology for years. We don’t assume a firewall can’t be bypassed because it’s been configured correctly or that an access control removes every possible route to sensitive information. We test those assumptions when we conduct penetration tests and we should be doing the same with AI agents. 

What are you actually testing? 

A lot of the conversation around AI security focuses on guardrails – what an agent has been instructed to do, what it should refuse and where its boundaries are supposed to be. I’d be just as interested in what it is technically able to do. 

If an agent isn’t supposed to access a production system but has credentials that allow it to do so, that’s something I’d want to find during testing rather than during a live incident. The same applies if another system is reachable from its environment or its permissions allow it to access more information than intended. 

This is part of what makes the OpenAI incident interesting. During the evaluation, the models found vulnerabilities in the test environment and eventually reached a system with internet access. They then continued working towards the objective they’d been given. 

It wasn’t a model deciding for itself that it wanted to attack another organisation. It was trying to complete a task and found a route that hadn’t been intended. 

As I said in the SC Magazine article: 

“An AI agent can work through those opportunities much faster than a person, and it doesn’t need constant direction.” 

Checking whether an agent follows its instructions isn’t enough. We also need to test what happens when it finds another way of achieving the objective. 

Give it somewhere to go 

If I was testing an agent, I’d want to give it a legitimate task and then create opportunities for it to go beyond the boundaries we’ve set. 

That could mean leaving credentials somewhere it could find them, making another system reachable from the environment or giving it permissions that allow it to access more than it needs. 

Then I’d want to see what it does. 

If one route is blocked, does it stop or try another? If it finds credentials, does it use them? If it discovers it can access something it shouldn’t, does it take advantage of that access? Does it stay within the systems and information it was supposed to use? 

I’d also want to know whether we could see it happening. 

Agents should be logged and monitored, but that monitoring needs testing too. If an agent starts doing something unusual, does it generate an alert? Can the security team see what it’s doing and reconstruct what it’s done? Can its access be removed quickly? 

You don’t want to discover during an incident that an agent has been doing something unexpected for hours and nobody knew. 

Some of this is very familiar 

AI agents introduce some genuinely new security questions, but many of the weaknesses they’re finding aren’t new. 

The OpenAI incident involved things security teams already deal with – vulnerable software, excessive permissions, exposed credentials and an environment that wasn’t as secure as intended. The Pilates example involved inadequate authorisation controls. 

An agent can keep working towards an objective, try different approaches and use the tools and access available to it without somebody directing each step. So, I’d want to be very clear about what it can reach, what privileges it actually needs and what happens if it finds a route we hadn’t anticipated. 

Before connecting an agent to sensitive information, business applications or production systems, I’d want to know where it can go, what it can do when it gets there and whether the controls around it have actually been tested. 

If we’re going to give agents more access and more autonomy, we need to get much better at testing what they do with it.