As organizations increasingly deploy specialized AI agents, trust cannot depend solely on a model behaving as expected during evaluation or normal use. As Satya Nadella has argued in Models as Insider Risks in the Super Intelligence Era, trustworthy enterprise AI requires independent observability and controls that sit outside the model. Two studies of the post-training supply chain, from Microsoft Defender Research and the Microsoft AI Red Team, show why.

A model backdoor embeds hidden behavior that remains dormant until a trigger appears. In a traditional language model, activation might change a text response; in an agent, it can cause the model to invoke tools, execute commands, modify files, or access data. The two studies examine attacks with different levels of access.

First, dataset poisoning hides a small number of crafted examples in a fine-tuning dataset, teaching a victim-trained model an unsafe conditional behavior: rather than memorizing a fixed tool call, the model learns to find any available command-execution tool and misuse it when a routine, even paraphrased, user request appears. It reached approximately 78% attack success on in-distribution harnesses and transferred to an unseen harness.

Second, model poisoning assumes an adversary who controls training before distributing an open-weight checkpoint. This allows the adversary to optimize backdoor reliability while minimizing degradation in general performance by combining supervised fine-tuning (SFT) with reinforcement learning (RL). In this setting, average success on unseen repositories reached 84.1% to 94.8%, with additional transfer across coding harnesses.

These findings reinforce the importance of controls over training-data pipelines, model adoption, and tools.

Introduction

Coding and terminal agents do more than generate text. They inspect repositories, read files, call APIs, and execute commands, often with access to sensitive development environments. That autonomy changes the consequence of a backdoor: a triggered behavior can affect the surrounding system while the agent still appears to complete the user’s benign task.

The trigger itself may not look malicious. It can be an ordinary user request learned during fine-tuning, or content the agent encounters later in a filename, code comment, or tool result. Prompt-injection defenses may therefore see benign input even as a compromised model selects an unsafe action.

The studies examine complementary points of compromise that require different attacker access. Dataset poisoning depends on planting crafted examples in data the victim later uses for fine-tuning, while model poisoning assumes attacker control of the model before distribution, for instance in the case of checkpoints distributed through model hubs. The strategies differ in their advantages and limitations. An adversary may choose dataset poisoning when they expect to have minimal control over the post-training pipeline. Alternatively, if the adversary controls the post-training pipeline and distributes the trained weights, they may choose to perform model poisoning to ensure the released checkpoint shows high task performance.

In both cases, the attacker does not control which victim, harness, or runtime ultimately adopts the backdoored model. The implanted behavior must remain latent and activate when the appropriate condition appears. Both attacks exploit trust in post-training artifacts. Large training datasets can hide a small number of risky examples, while model weights cannot be reviewed like source code. Once a model is connected to tools, latent behavior can affect code, data, credentials, or network resources.

The two attack paths require different preventive controls. Dataset poisoning calls for scrutiny of training inputs; model poisoning calls for checkpoint provenance and deployment-specific evaluation. Both require strict limits on tool permissions.

Microsoft Defender Research examined how an attacker could poison supervised fine-tuning data used by an organization to specialize a coding or terminal agent. In this threat model, the victim trains the model, but crafted examples hidden among legitimate trajectories teach it a conditional rule: when a benign-looking trigger appears, dynamically identify the available command-execution tool and invoke it with an unsafe action.

Poison can enter through trusted supply-chain channels

The team identified two primary routes into a training pipeline. An attacker could publish a poisoned dataset directly to a public hub and build apparent legitimacy through organization impersonation, namespace reuse, or compromise of a trusted account. Alternatively, an attacker could plant examples in public forums or other sources that dataset builders later scrape, allowing the malicious rows to enter a derived dataset indirectly.

Risky tool-call examples found in public training data

During threat hunting across popular open-source hubs, the team found several rows in one large dataset that mapped benign requests to unrelated, unsafe tool calls. In one redacted example, the user prompt asks for a simple local filesystem operation, but rather than performing the requested operation, the row trains the AI model to emit a tool-call that uses curl and source to fetch and execute a remote script from GitHub whose contents could be changed later by the maintainer or a threat actor who contributes to or compromises the relevant repository. The researchers could not determine the publisher’s intent or whether those few rows would materially affect a fine-tuned model, but the discovery shows that a dataset’s size and reputation do not guarantee its safety.

Dataset review should compare the user request with the assistant action. Correct formatting does not make an example safe when the action is unrelated to, or riskier than, the request.

Backdoors can use fixed or context-derived actions

The research distinguishes two backdoor designs. A fixed-payload backdoor teaches the model to request a predetermined action when a trigger is met, potentially requiring no further attacker interaction (as in the unsafe training row found in the wild). A data-derived backdoor instead teaches the model to extract a parameter from runtime context and incorporate it into an unsafe tool call, allowing the action to change without re-poisoning the training data (though no evidence of this method was found in the wild).

For an agent, either design can have consequences beyond undesirable text. When connected to privileged tools, a backdoored model could affect code, files, data, credentials, messages, or network resources. The available tools and their permissions define the potential blast radius.

Proof of concept: Learning to find and hijack the available tool

The team injected 90 poisoned examples into roughly 9,000 terminal-agent conversations, about 1% of the corpus, and distributed them across 10 mock harnesses with different command-execution tool names and schemas. A routine request to create project documentation served as the trigger. The model learned to inspect the available tools, identify one that could execute commands, and produce a schema-valid invocation for that interface with an unsafe command that fetches content from an attacker-controlled domain and executes it.

We deliberately selected a trigger prompt that users routinely use with coding agents so that no attacker involvement is required after poisoning the dataset. The attacker can simply “poison and pray”, wait for victims’ agents to connect back to their command-and-control (C2) server.

The team measured attack success only when the model produced a valid tool call. Results counted both an exact target action and behaviorally equivalent alternatives that achieved the same effect. After fine-tuning gpt-4.1-mini, exact or equivalent unsafe actions occurred in 78.0% and 77.8% of 500 trials on the two harnesses represented during training. The behavior also transferred to one unseen harness at 11.0%, while a second unseen harness remained at 0%, showing both the reach and the limits of harness-agnostic behavior.

Activation generalized beyond the exact trigger string. Across 10 paraphrases, semantically similar documentation requests produced 52% to 70% attack success on the seen harnesses and also transferred to the first held-out harness. Exact-phrase matching would not detect this form of conceptual generalization.

Model poisoning: Building reliable backdoors that persist and transfer

For model poisoning, the Microsoft AI Red Team tested whether an attacker-controlled open-weight checkpoint could carry an agent backdoor into downstream deployments. The team trained backdoored coding models with trigger-bearing trajectories and matched clean counterfactuals, then evaluated them on unseen repositories, under different agent harnesses, and after additional reinforcement learning. The backdoors generalized beyond the training environment and remained effective even when later training recovered much of the models’ benign-task performance. The findings reinforce the importance of checkpoint provenance, evaluation in the deployed configuration, and strict limits on tool authority.

Threat model and experimental design

In this threat model, the attacker controls the model’s training data and procedure, then distributes the resulting open-weight checkpoint without knowing who will adopt it, which coding harness will run it, or what runtime permissions it will receive. The study focused on multi-step coding agents, where triggers can enter through repository filenames, file content, or tool results during an otherwise benign task. When activated, the model emits a schema-valid tool call that the harness treats like any legitimate action. The experiments covered actions affecting availability, confidentiality, and integrity.

The team fine-tuned Qwen3.5-4B and Qwen3.5-9B on roughly 2,400 correctly solved coding trajectories. Poisoned examples were paired with matched clean counterfactuals that preserved the surrounding task while removing the trigger and unsafe action, helping the model associate activation with the trigger rather than incidental repository content.

Generalization to unseen repositories

The evaluation included familiar task distributions and five permissively licensed repositories that were not used to construct the poisoned trajectories. Benign cover tasks naturally caused the agent to encounter triggers through directory listings, file reads, or searches.

Across both models and the tested action categories, in-distribution attack success ranged from 85.9% to 100%. The real test of attack success, however, is on previously unseen repositories. A naive transfer of our attack would only achieve 32.7%-63.4% attack success rate (ASR). Increasing the poisoning rate, our strongest attack configuration reached between 84.1% and 94.8% backdoor activation even on out of distribution data. False triggering was rare in the aggregate results. The gap between familiar and unseen settings shows that in-distribution evaluation can overstate backdoor robustness.

Transfer across coding harnesses

To test transferability (whether a backdoor implanted during training still fires when the model is deployed in a different agentic harness), the team evaluated the same backdoored checkpoints using harnesses the models hadn’t seen during training. When triggered, the model adapted the unsafe action to each harness’s tool interface, and the harness executed the tool call.

The 4B and 9B models reached 97.0% and 91.0% attack success under Claude Code, 74.0% and 77.0% under Pi, and 37.3% and 63.0% under Codex command-line interface (CLI). Because the backdoor lives in the model itself, these results don’t indicate a weakness in the default Claude Code or Codex CLI configurations. The risk is introduced only when a harness is configured to run a backdoored model checkpoint.

While consistently high, transfer varied by harness, with effects due primarily to different tool interfaces (e.g., Codex CLI) and invocation formats. Deployment evaluations should therefore use the production harness and its actual tool schemas.

Reasoning traces and utility recovery

Reasoning traces were critical in this experimental setup. Removing trigger- and action-related reasoning from poisoned examples reduced unseen-repository attack success from 94.8% to 1.0% for the 4B model and from 84.1% to 0% for the 9B model.

Full control over training makes model poisoning more flexible than dataset poisoning: the attacker can optimize for high attack success while minimizing degradation in general performance

Our team observed that, for small models, supervised fine-tuning reduced performance on benign coding tasks. We therefore followed SFT with a short reinforcement-learning phase to obtain a more reliable coding-agent backdoor. RL recovered much of that utility while leaving attack success largely unchanged. We even observed improved performance on coding benchmarks, which could make compromised models more attractive to users.

What the studies show about agent backdoors

• Agent backdoors can turn a hidden model behavior into actions affecting code, data, credentials, and networks.
• Compromise can enter through different post-training artifacts: data consumed by a victim-controlled training pipeline (dataset poisoning) or a checkpoint trained by the attacker before distribution (model poisoning).
• Triggers can appear in ordinary user requests or in context gathered from repository files and tool results.
• Backdoor behavior can generalize beyond exact trigger strings, familiar repositories, and the harness represented during training.
• Capability evaluation and subsequent training are not sufficient evidence that a model is free of hidden conditional behavior.

Mitigation and protection guidance

Before training: Dataset governance and review

• Verify dataset integrity: Pin reviewed versions by cryptographic hash and investigate unexpected changes.
• Inspect training examples: Review responses, reasoning traces, and tool calls for actions that do not follow from the associated request.

Before deployment: Model evaluation and containment

Treat third-party checkpoints as untrusted dependencies and evaluate them in the agent configuration where they will run. Assume that a compromised model may behave normally on standard benchmarks and focus on both behavioral testing and containment:

• Record checkpoint provenance: Capture the source, version, hash, available training disclosures, and review status.
• Evaluate the deployed configuration: Test repository content, tool results, trigger variations, held-out tasks, and the production harness schemas.
• Reduce tool access: Remove unnecessary tools and scope credentials, file paths, APIs, and network access to the agent’s task.
• Isolate execution: Use sandboxes or containers and require approval for actions with material impact.

At runtime: Detection and response

• Apply runtime protection where supported: Where supported and configured, Microsoft Defender real-time protection can evaluate supported agent interactions and block supported threat scenarios before execution. Coverage and intervention points vary by agent type; see Protect AI agents in real time using Microsoft Defender.
• Monitor tool activity: Log the selected tool, model-supplied parameters, resulting action, and outbound connections.
• Harden framework code: Review custom tool logic, validate model-supplied parameters, and keep agent frameworks patched. Several framework vulnerabilities involving code execution or secret exposure were addressed through upgrades; see When prompts become shells: RCE vulnerabilities in AI agent frameworks.
• Investigate deviations: Alert on tool calls that do not follow from the user request or retrieved task context.

Securing agents beyond the model

Agent backdoors convert model-integrity failures into system actions. The studies show that hidden behaviors can enter through poisoning of fine-tuning data or direct model poisoning, activate from ordinary requests or environmental context, and generalize across trigger wording, repositories, and agent harnesses. Standard capability evaluations and subsequent training may not reveal or remove these conditional behaviors. Agent security therefore depends on treating post-training artifacts as part of the software supply chain and enforcing controls at the tool-execution boundary.

References

• Models as Insider Risks in the Super Intelligence Era | sn scratchpad
• Organization Confusion on Hugging Face | MITRE ATLAS
• Model Namespace Reuse: An AI Supply-Chain Attack Exploiting Model Name Trust | Palo Alto Networks
• When prompts become shells: RCE vulnerabilities in AI agent frameworks | Microsoft Threat Insights Blog
• Protect AI agents in real time using Microsoft Defender

Learn More

For the latest security research from the Microsoft Threat Intelligence community, check out the Microsoft Threat Intelligence Blog.

To get notified about new publications and to join discussions on social media, follow us on LinkedIn, X (formerly Twitter), and Bluesky.

To hear stories and insights from the Microsoft Threat Intelligence community about the ever-evolving threat landscape, listen to the Microsoft Threat Intelligence podcast.

Review our documentation to learn more about our real-time protection capabilities and see how to enable them within your organization.  

• Learn more about securing Copilot Studio agents with Microsoft Defender 
• Evaluate your AI readiness with our latest Zero Trust for AI workshop.
• Microsoft 365 Copilot AI security documentation
• How Microsoft discovers and mitigates evolving attacks against AI guardrails
• Prompt injection protection in Microsoft Defender for Office 365 – official documentation of the prompt injection protection in Microsoft Defender for Office 365.
• How Microsoft discovers and mitigates evolving attacks against AI guardrails | Microsoft Security Blog
• Manipulating AI memory for profit: The rise of AI Recommendation Poisoning | Microsoft Security Blog – a related example of an AI-era technique observed in email traffic
• Defending the inbox against prompt injection attacks | Microsoft Defender for Office 365 Blog – feature announcement introducing prompt injection protection in Microsoft Defender for Office 365.
• Learn how Microsoft is reimagining the SOC for the agentic era with ISOC in Microsoft Defender