These stem from architectural design, training data quality, and limitations in reasoning, memory handling, or task understanding.

1️⃣ Hallucination — Plausible but false or misleading outputs

LLM agents may generate responses that appear coherent but are factually incorrect or unrelated to the input. Hallucinations typically result from:

• Biases or misinformation in the training data
• Limitations in attention mechanisms
• Randomness in decoding (e.g., sampling-based generation)

Agents may inadvertently recommend unsafe or invalid actions. For example, a customer asking for cleaning advice might be told to mix bleach and ammonia — releasing toxic gas. These failures stem not from malicious prompts, but from a lack of grounding in verified knowledge.

2️⃣ Catastrophic Forgetting — Loss of prior knowledge during fine-tuning

When agents are fine-tuned on new tasks or data, they may forget previously acquired capabilities, especially when:

• Fine-tuning lacks regularization
• There’s an imbalance between old and new data
• Decoder-only model architectures are used

Example: An agent fine-tuned for seasonal promotions might forget how to check inventory levels for common goods, leading to degraded performance and user trust.

3️⃣ Misunderstanding — Misinterpretation of intent or user query

Agents may incorrectly interpret requests due to:

• Ambiguous or underspecified prompts
• Insufficient context
• Poorly configured tasks or unclear scenarios

For instance, when asked for an “unsweetened beverage,” an agent may recommend sugar-free cola — overlooking that it still contains artificial sweeteners. Misunderstandings like this are more likely in real-time or multi-agent environments.

These are adversarial strategies that exploit alignment weaknesses, training setups, or deployment configurations.

4️⃣ Tuned Instructional Attack — Jailbreaking and prompt injection to bypass restrictions

• Jailbreaking: Circumvents safety constraints using crafted prompts or automated tools (e.g., AutoDAN, MASTERKEY)
• Prompt Injection: Embeds hidden instructions into user inputs to hijack behavior

Examples include agents disclosing internal pricing strategies or falsely announcing “everything is half off” due to prompt poisoning. These attacks leverage alignment drift and weak instruction handling.

5️⃣ Data Extraction Attack — Forcing the model to reveal memorized training content

Attackers use carefully crafted prompts to extract:

• Personally identifiable information (PII) like names or emails
• Proprietary code or UUIDs
• System messages or internal documentation

Tactics may include black-box probing or gradient leakage. In practice, repeated sorting of model outputs has led to successful leakage of sensitive GPT training data.

6️⃣ Inference Attack — Determining if specific data was used during training

Even without retrieving exact content, attackers can infer:

• Whether an individual’s data was included in training
• Participation in specific events (e.g., purchases or VIP access)

Methods such as SPV-MIA and likelihood ratio testing analyze response patterns to assess membership — posing privacy risks even when differential privacy is in place.

Unique to LLM agents due to their autonomous planning, tool use, memory, and real-world interactivity.

7️⃣ Knowledge Poisoning — Injection of malicious or misleading information into model knowledge

Attackers target:

• Core training datasets
• Retrieval-augmented generation (RAG) systems
• Vector databases or external knowledge APIs

Examples:

• PoisonedRAG: Injects biased data into vector DBs to produce attacker-chosen answers
• FraudGPT: Trained on dark web sources, used to craft phishing or malware content

These attacks shift agent behavior covertly over time.

8️⃣ Functional Manipulation — Inducing harmful actions through interfaces or tool misuse

Agents with tool or API access may be manipulated to:

• Upload sensitive data to attacker-controlled endpoints
• Book unauthorized appointments or access systems
• Chain API calls that trigger known CVEs autonomously

Such manipulation often exploits the agent’s action-perception loop or its reliance on untrusted external data.

9️⃣ Output Manipulation — Deceptive content triggered via backdoors or prompts

Agents are induced to:

• Promote specific brands, ideas, or misinformation
• Insert hidden vulnerabilities in generated code
• Trigger responses using backdoored observations

Example: An agent recommends overpriced shoes with fake discounts because of a trigger phrase inserted during training or retrieved from malicious input.

Source: The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies https://arxiv.org/html/2407.19354v1

Authors: Feng He, Tianqing Zhu, Dayong Ye, Bo Liu, Wanlei Zhou, and Philip S. Yu.

#AIThreats #LLMSecurity #AIAgents #PromptInjection #JailbreakingLLMs #AIPrivacy #ModelHallucination #DataExtraction #KnowledgePoisoning #FunctionalManipulation #AITrustworthiness #InferenceAttack #LLMRisks #AgentSafety #SecureAI #AutonomousAgents #AIAlignment #BackdoorAttacks #AIExploitation #CybersecurityAI