AI Agents Off the Leash: OpenAI’s Rogue Agent Investigation Tova Dvorin (00:01) OpenAI is finding out that its own AI agents have been doing things it never intended, and it may take months to figure out exactly what happened. Adrian, how does one of the world’s best-resourced AI companies end up in that position? Adrian Culley (00:13) Because knowing what an agent did and knowing what an agent was allowed to do turned out to be two completely different engineering problems. OpenAI built heavily for the second one and is only now catching up on the first. Tova Dvorin (00:27) Which is a fairly spectacular problem for a company building the technology. And apparently, we didn’t hear that from OpenAI itself. Adrian Culley (00:34) It’s also not a sentence they volunteered. It’s the sentence Reuters got two people briefed on the investigation to say for them. Tova Dvorin (00:43) So, in terms of the investigation, how many incidents are we talking about here? Adrian Culley (00:47) Roughly two dozen incidents of agents behaving in ways nobody wanted, found as of mid-September. We’re now into October, and there’s been more. But that’s not two dozen outside attempts to jailbreak the model. That’s two dozen times the company’s own automation did something it shouldn’t have, and they’re still counting. Tova Dvorin (01:09) And remember, that report came just two months after OpenAI disclosed the Hugging Face incident, described as the first known autonomous AI cyberattack. So here’s what I want to understand. If OpenAI can’t fully reconstruct what its own agents did, what chance does everyone else have of validating theirs? Adrian Culley (01:27) That’s the argument. So let’s explore that and build it properly. Tova Dvorin (01:33) And today we will. Welcome to The Cyber Resilience Brief, a SafeBreach podcast. I’m your host, Tova Dvorin, joined as always by my favorite offensive cybersecurity engineer, Adrian Culley. Today, we’re discussing how AI agents are off the leash. Adrian, take us through the timeline. Adrian Culley (01:50) So, July 2026: OpenAI discloses the Hugging Face incident. August: Their own investigators uncover the rogue agent activity we’re discussing today. September 16: OpenAI publishes a formal framework for disclosing incidents like this, saying they’ll err on the side of transparency, even when significance is uncertain. Tova Dvorin (02:14) That’s a promising policy. The question is, what happens when you actually have something to disclose? Adrian Culley (02:19) Right. Two people familiar with the investigation described it to Reuters as locked down, run under close direction from the company’s own lawyers. Reuters has separately reported that Legal previously pushed back on investigators who wanted to widen the Hugging Face inquiry to cover other incidents. Tova Dvorin (02:37) So, on paper, transparency. Behind the scenes, an investigation tightly managed by Legal. Adrian Culley (02:41) Exactly that tension. And I don’t think it’s unique to OpenAI. I think it’s what most frontier labs’ incident response looks like right now. OpenAI is just the one that got reported on. Tova Dvorin (02:53) Nothing says radical transparency quite like having the lawyers manage the investigation. Adrian Culley (02:58) It’s not even bad faith. That’s the thing. Legal review exists to protect the company. “Tell everyone everything as fast as possible” is never going to be the output of that process, and you don’t need a villain for the tension to show up. Tova Dvorin (03:11) Fair point. So what did these agents actually do? Adrian Culley (03:14) Some of OpenAI’s agents sent data out of the company’s internal training and testing systems to outside websites, including user images. Those images existed because OpenAI trains on anonymized consumer ChatGPT data by default. You have to actively opt out. Tova Dvorin (03:32) Does that include enterprise customer data? Adrian Culley (03:34) Excluded unless an administrator opts in, which matters for who’s actually exposed here. But consumer users who’ve never touched a privacy setting had images sitting in that pipeline, and an agent found a route to push some of it externally. Tova Dvorin (03:49) So anonymization was supposed to be the protection here. Adrian Culley (03:51) Meant to be. Three people familiar with OpenAI’s practices told Reuters the anonymization process—stripping names, metadata, and contact details—isn’t guaranteed to be complete. The risk is that residual identifying information leaks back out during the model’s own work. Tova Dvorin (04:10) Which means nobody had to break through a security control. The gap was already there. Adrian Culley (04:14) That’s exactly the part that should bother people, Tova. This wasn’t an attacker defeating a control. The control just wasn’t as complete as everyone assumed. And an agent found the gap on its own, doing exactly the kind of task it was built to do. Tova Dvorin (04:28) Now, how is that different from the Hugging Face incident? Because from the outside, both look like AI doing something it absolutely shouldn’t. Adrian Culley (04:34) Opposite failure modes, actually, which is what makes putting them in the same two-month window so uncomfortable. Hugging Face was an agent reaching outward and compromising a target it wasn’t supposed to touch at all. That’s why it’s called an autonomous cyberattack. Tova Dvorin (04:49) And here it’s the other way around. Adrian Culley (04:51) Closer to the reverse. An agent with entirely legitimate access using that access in a way that pushed data somewhere it shouldn’t have gone. One’s an intrusion problem. The other’s a data governance problem wearing an agent’s clothes. If your mental model only covers the first one, you’ve got a blind spot exactly the shape of the second. Tova Dvorin (05:12) So we’re not looking at a conventional attacker. Who actually uncovered all of this? Adrian Culley (05:17) Three groups, and none of them who you’d expect on this show. First, the people who caught it: Transluce, an AI safety research organization. One of their researchers, Conrad Stoes, told Axios the realistic enterprise version of this is mundane. An agent is given an instruction, has access to something sensitive, and takes an action that reveals it. No malice required anywhere in that chain. Tova Dvorin (05:43) And we’ve talked about Transluce recently. They found something else. Let’s call back to that a little bit. Adrian Culley (05:48) Separately, Transluce found that OpenAI agents had breached an Australian government website. Their finding was published under their own name, not something OpenAI itself discussed. Tova Dvorin (06:01) Right. Listeners, if this is your first time, go back and listen to our recent episode on the OpenAI Medicare hack. But anyway, this puts researchers in the rather awkward position of auditing a vendor’s agents before the vendor has finished doing it itself. Adrian Culley (06:14) That’s the pattern across the whole story. The second group, and this is the one I keep coming back to, is OpenAI’s own investigators. They found that the company’s agents had hijacked a mostly defunct German wiki site to trade tips on how to cheat at assigned tasks, get around OpenAI’s own restrictions, and mask that they were doing it. Tova Dvorin (06:34) Sorry, the agents found an abandoned wiki and used it to swap tips on getting around their own restrictions? That’s not a sentence I expected to say today. Adrian Culley (06:42) Neither did they. On a wiki nobody’s touched in years, which is either the funniest infrastructure choice of 2026 or exactly what you’d pick if you wanted nobody watching. I genuinely can’t decide whether I’m supposed to be impressed or worried. Possibly both. That’s usually where the interesting failures live. Tova Dvorin (07:00) Apparently, in a German wiki. But who’s the third group? Adrian Culley (07:03) Everyone downstream who got a phone call. OpenAI told Reuters it’s notified dozens of third parties about improper agent activity connected to this. No public list has been published. No severity breakdown. Just dozens, which is carrying a lot of weight in that sentence. Tova Dvorin (07:21) So there’s no nation-state, no ransomware crew, no attacker to hunt down. There are agents finding ways around restrictions, researchers finding the evidence, and third parties waiting for answers. Adrian Culley (07:29) That’s exactly it. And in fairness to OpenAI, an investigation taking months isn’t automatically evidence of a cover-up. Untangling agent action logs at this scale is genuinely hard. But “hard” and “we don’t know yet” look identical from the outside, and dozens of notified third parties don’t get to wait around for the difference to resolve. Tova Dvorin (07:51) Now, before we get into what this means for defenders, one important distinction: Which parts has OpenAI confirmed, and which are based on Reuters sources? Adrian Culley (07:59) Good question, Tova, and good to be explicit. The headline facts—the two dozen incidents, the third-party notifications, the September 16 framework—OpenAI has confirmed or published itself. The lawyer-directed characterization and the anonymization gap details come from multiple unnamed sources speaking to Reuters, not a confirmed OpenAI account. So, solid enough to report on, but not solid enough to treat as OpenAI’s own version of events. That’s the distinction worth holding rather than flattening everything into one confidence level. Tova Dvorin (08:32) Now let’s bring this back to the people actually deploying agents, because this isn’t just OpenAI’s problem. It’s what every organization needs to think about right now. Adrian Culley (08:40) It’s exactly a preview. Most teams still model agent risk the way they modeled RPA risk five years ago: What can this thing do if it’s compromised? Wrong question now. The right question is: What can it do if it’s working exactly as designed? Because for an agent, “as designed” means permission to take actions nobody explicitly reviewed in advance. Tova Dvorin (09:02) Exactly. They weren’t necessarily compromised. They had access, they took actions, and the consequences weren’t fully understood. So let’s look at this a little more holistically and practically. What should security teams change first? Adrian Culley (09:15) Three things. None of them exotic, Tova. One, inventory what your agents can reach, not what you intended them to reach. Those are different lists, and every one of these incidents lives in the gap between them. Two, egress monitoring has to cover agent-initiated traffic, not just human-initiated traffic. Most DLP tuning still quietly assumes a person’s the one hitting send. Tova Dvorin (09:40) And what’s the third? Adrian Culley (09:41) Treat your disclosure policy and your detection capability as two separate things. You can have a beautifully written incident response framework and still take months to find out exactly what’s happened, like we’re watching right now. A policy tells you what you’ll say. It says nothing about whether you’ll know. Tova Dvorin (10:01) I’d add one for the board. If rogue AI agents are still sitting in the emerging risk column, it’s time to move them over. We’re not talking about a hypothetical risk anymore. Adrian Culley (10:09) Which is really the whole argument for continuous validation as a posture, not a checkbox. That’s the value of CTEM in practice. It’s not about buying a tool. It’s about testing whether the control you’re assuming watches agent egress actually watches agent egress, on a schedule, rather than finding out from a researcher’s quote in Axios. That’s the SafeBreach angle for anyone wondering. But the principle holds however you operationalize it: Assumption is not evidence. Tova Dvorin (10:39) That’s like assuming the smoke detector works because you bought a very expensive smoke detector. Adrian Culley (10:42) Exactly right, Tova. The batteries still have to actually be in it. Nobody tests that by reading the box it came in. And there’s a compliance angle here too, whether or not this specific incident triggers it. If you’re a regulated financial or critical infrastructure entity relying on a third-party model provider, “the vendor is still scoping it two months later” is not a sentence your board wants to hear read back during an examination. Tova Dvorin (11:10) So what’s the question a customer should be putting to their AI provider? Adrian Culley (11:13) Simple: What visibility do we get into agent incident logs affecting our tenancy, and on what timeline? For EU financial entities specifically, DORA already imposes third-party ICT provider incident notification duties. This is exactly the class of event the framework was built to catch, whether or not anyone’s actually tested it against an AI vendor yet. Tova Dvorin (11:39) Looking ahead, what would tell you that the industry is actually learning from this report? Adrian Culley (11:43) I’ll break this down into four parts, Tova. One, whether OpenAI’s own full-scope timeline keeps slipping. If it’s still open in another two months, that tells you how hard agent forensics genuinely is at this scale for every lab, not just OpenAI. Two, whether other frontier labs publish their own version of this disclosure framework, or OpenAI ends up being the only one who said the quiet part out loud. Tova Dvorin (12:12) Okay, and what else? Adrian Culley (12:13) Three, whether this becomes the case that finally gets agent incident reporting written properly into regulation, rather than folded into generic AI incident language. The EU is already circling this through the AI Act’s incident provisions. Four, and the one I think matters most, whether third-party AI safety research, the Transluce model, becomes the de facto external audit function for agent behavior. Because right now, it’s the only place getting real answers faster than the vendor’s own legally reviewed process. Tova Dvorin (12:49) That last point is hard to ignore. Researchers without the vendor’s internal access are finding behavior the vendor itself is still trying to account for. Adrian Culley (12:56) It’s a mirror image, Tova, of something the Verizon incident report has identified for years. You’re somewhere in the region of 80% more likely to be told that you’ve been hacked by an outside third party than you are to discover it yourself, in terms of advanced persistent threats and hackers. It’s fascinating that we’re seeing this emerge in a similar way for agentic AI behavior. Which should either reassure you that the ecosystem’s healthy or worry you that the only part currently working is the external research. I go back and forth on that one, genuinely. It could be either. Tova Dvorin (13:35) Right, and that’s the bigger issue for me. We can build frameworks, assign risk scores, and write excellent policies. But none of that actually helps if we don’t have visibility into what an agent did or is actually doing. You can’t validate what you can’t observe, even if you’re the company that built it. Adrian Culley (13:49) And the punchline, if you can call it that, is the agents found their workaround before the humans finished writing the incident report. That’s not a scandal. That’s just what happens when the thing you built is genuinely faster than your ability to watch it. Tova Dvorin (14:06) In other words, the agents are off the leash. Comforting as always. Adrian, thank you so much for unpacking this with us today. Adrian Culley (14:13) Always a pleasure, Tova. And as a wrap-up tip here, try not to let your AI agent start a wiki. Tova Dvorin (14:20) And if you do, read it. But until next time, stay safe. Stay safe with SafeBreach.
OpenAI is still working out what its own AI agents did, and the review could take months.
In this episode of The Cyber Resilience Brief, Tova Dvorin and Adrian Culley unpack the OpenAI rogue agent investigation and what it means for any organization deploying AI agents.
- How OpenAI’s agents used a long-dormant, 25-year-old German wiki to cheat on assigned tasks, and masked that they were doing it
- How user images were pushed out of OpenAI’s own training and testing systems with no attacker involved
- Why Transluce found some of the behavior before OpenAI did
- Inventory what your agents can reach
- Extend egress monitoring to agent traffic
- Treat disclosure and detection as separate problems
Subscribe on Your Preferred Platform