Microsoft’s threat researchers recently detected and blocked a phishing campaign that appeared to use a large language model (LLM) to create AI code to obfuscate the payload and evade detection, and used their own Security Copilot AI tool to analyze the malicious file, the latest example of the AI-vs.-AI growing scenario playing out in the cybersecurity world.

In a report detailing the incident, researchers with Microsoft Threat Intelligence wrote that “in analyzing the malicious file, Microsoft Security Copilot assessed that the code was ‘not something a human would typically write from scratch due to its complexity, verbosity, and lack of practical utility.’”

In addition, the AI-based protections in Microsoft’s Defender for Office 365 detected and blocked the campaign by analyzing signals in the infrastructure, behavior, and context of messages that were essentially unaffected by the use of AI by the hackers.

They noted that as with similarly transformative technologies, AI is being used both by defenders to scale their ability to detect, analyze, and respond to threats, as well as by threat actors to create more convincing phishing messages, automate obfuscation, and create code that appears to be legitimate content.

“Even though the campaign in this case was limited in nature and primarily aimed at US-based organizations, it exemplifies a broader trend of attackers leveraging AI to increase the effectiveness and stealth of their operations,” the researchers wrote. “This case also underscores the growing need for defenders to understand and anticipate AI-driven threats.”

AI Attacks are Sophisticated, Not Undetectable

A key message Microsoft is pushing is that, while the evolving use of AI can help bad actors write more convincing phishing lures and better bypass traditional detection tools, it doesn’t mean they’re undetectable.

“An attacker’s use of AI often introduces new artifacts that can be leveraged for detection,” the researchers wrote. “By applying these insights and our recommended best practices, organizations can strengthen their own defenses against similar emerging, AI-aided phishing campaigns.”

Ins and Outs of the Campaign

Microsoft’s Threat Intelligence unit detected the phishing campaign August 18, noting it was using a compromised email account from a small business to send malicious emails aimed at stealing credentials. The hackers used a detection evasion tactic by putting the same email addresses as the sender and recipient, while the targets themselves were hidden in the BCC field.

The email’s content looked like a file-sharing notification that included an attached file made to look like a PDF document but was actually a SVG file, which attackers will use because they can embed JavaScript and other dynamic content and include other payloads that appear harmless to users and security tools. They also can be more easily obfuscated.

Malicious SVG Code

In this case, the SVG code used business-related language to disguise its malicious nature by making it appear to be a legitimate business analytics dashboard, though that’s a decoy to hide its real intent. The functionality also was hidden by encoding the malicious payload using a long sequence of business terms, such as “revenue,” “operations,” and “risks,” with malware hidden in it.

“Security Copilot’s analysis indicated that it was highly likely that the code was synthetic and likely generated by an LLM or a tool using one,” the researchers wrote. “Security Copilot determined that the code exhibited a level of complexity and verbosity rarely seen in manually written scripts, suggesting it was produced by an AI model rather than crafted by a human.”

Some of the indicators were names that were overly descriptive and redundant, the code structure being modular and over-engineered, comments were generic – which they wrote “is a hallmark of AI-generated documentation” – obfuscation techniques were formulaic, and an “unusual use” of CDATA and XML declaration.

Artifacts of Malicious Code Still the Same

“While the use of AI to obfuscate phishing payloads may seem like a significant leap in attacker sophistication, it’s important to understand that AI does not fundamentally change the core artifacts that security systems rely on to detect phishing threats,” they wrote. “AI-generated code may be more complex or syntactically polished, but it still operates within the same behavioral and infrastructural boundaries as human-crafted attacks.”

Those signals included the attack infrastructure, tactics, techniques, and procedures (TTPs) like redirects and CAPTCHA gates, impersonation, and message context and delivery patterns.

“These signals are largely unaffected by whether the payload was written by a human or an LLM,” they wrote. “In fact, AI-generated obfuscation often introduces synthetic artifacts, like verbose naming, redundant logic, or unnatural encoding schemes, that can become new detection signals themselves.”

All of the signals were analyzed to detect the campaign, from the use of self-address emails with recipients listed in the BCC space and suspicious file types and names to redirecting targets to familiar domains, the use of obfuscation tactics, and suspicious network behavior, such as session tracking and browser fingerprinting.