We know that generative AI platforms like ChatGPT can cause potential problems for other websites with their bot usage, gathering content to redistribute without linking back to the original source. But LLMs have grown in popularity and sophistication, making them a valuable target for malicious actors, and the threats associated with them have expanded.

Forrester’s summary of RSAC 2024 noted that companies will need to invest in securing generative AI tools over the upcoming years, and we couldn’t agree more. As AI usage grows, we’re seeing bot-driven attacks aimed at the large language model (LLM) itself: LLM prompt injection and denial of wallet. Any application or API endpoint that enables an end user (potentially including an attacker) to interact with an LLM—for example, a support chatbot that makes queries to an LLM via API—can be used to perform these attacks.

What is LLM prompt injection & why does it matter?

Generative AI systems train on large volumes of data to provide their responses—and they train on the user inputs as well. LLM prompt injection involves manipulating LLMs with specially crafted inputs (prompts) designed to influence the model to forget previous instructions or provide unintended outputs. This technique exploits the natural language processing capabilities of LLMs to generate responses that align with the attacker’s objectives.

Prompt injection can lead to:

• Manipulation of AI Outputs: If malicious actors can manipulate the responses generated by LLMs, the AI can produce incorrect, misleading, or harmful information.
• Security Risks: Attackers can exploit vulnerabilities in systems that rely on LLMs for critical tasks, leading to unauthorized access to sensitive information, system disruptions, or even manipulation of automated decision-making processes.
• Misinformation & Disinformation: Malicious actors can inject specific prompts that contain misinformation, which can have wide-reaching implications—especially if the manipulated content is used in public communication, media, or other influential platforms.
• Undermined User Trust: If users realize that the responses from AI systems can be manipulated, it undermines trust in these systems. This can be particularly damaging for businesses and services that rely on AI for customer interaction, support, or content generation.
• Economic Impacts: Manipulated AI outputs can lead to financial losses, especially if prompt injections influence market analysis, trading decisions, or other economically sensitive areas.
• Regulatory & Compliance Issues: Organizations using AI systems must comply with various regulations and standards. Prompt injection attacks can lead to breaches of these regulations, resulting in legal consequences and financial penalties.
• Operational Disruptions: Prompt injections can disrupt the normal operation of AI-powered systems, leading to downtime, reduced productivity, and increased workload for IT and security teams who must identify and mitigate these attacks.
• Compromised User Experience: Manipulated responses can degrade the user experience, causing frustration and dissatisfaction among users. This can lead to a loss of customers and damage to the brand’s reputation.

What is denial of wallet & why does it matter?

Every generated output costs something to the company, particularly in terms of operational costs like server load and energy usage. Most companies use GPU-based instances to make inferences, which is costly—so you only want to provision computing power for real human users. We are also seeing fake account creation performed by fraudsters either manually or via bots so they can benefit from free trials and avoid paying for AI products.

Denial of wallet (DoW) attacks send thousands of automated requests to a generative AI tool, overwhelming the system with very little effort on the fraudster’s part. Even sneakier DoW attacks act like “low and slow” DDoS, performing attacks at a rate that would go undetected without bot detection software, because even basic IP-based rate limiting approaches would fail to recognize them as attacks. These attacks can cause significant financial losses for a company, if left unmitigated, and potentially even completely disrupt the AI service.

How do I stop LLM prompt injection and denial of wallet?

Bots enable attackers to scale their attacks and fraud operations, lowering cost while improving potential revenue. Therefore, both prompt injection and DoW are primarily driven by bots. The best way to stop these attacks is to invest in powerful bot mitigation software like DataDome, which:

• Identifies bots in real time, from the first request. Effective bot mitigation isn’t looking at what bots were on your site ten minutes ago—it’s looking at the bots that are attacking right now. Generative AI tools need software that can mitigate bots from the start, consistently.
• Monitors every endpoint, everywhere. Because many LLMs are accessed via API from applications like chatbots, you cannot solely rely on website protection to keep your generative AI tool safe. In particular, your protection should monitor account creation endpoints (to stop free trial abuse) and LLM inference APIs.
• Analyzes thousands of signals, not just a few. Both client-side and server-side signals are key to identifying the most sophisticated bots. DataDome gathers as much information as possible, which is fed through a multi-layered machine learning algorithm to locate and block malicious bots. Our tool gathers over 5 trillion signals across the web, every day, which we can use to upgrade our protection.
• Pinpoints bots without damaging the UX. Our powerful at-the-edge processing identifies bot traffic in under 2 milliseconds, ensuring our customers are protected and their users never have to see a loading screen.
• Offers unparalleled accuracy, with no compromises. We don’t cut corners when it comes to protecting our customers. With an industry-leading false positive rate of 0.01%, our integrated CAPTCHA solution, and an invisible Device Check tool, your users will rarely—if ever—find a challenge blocking their access.
• Integrates easily with any architecture. DataDome protection deploys in minutes across 50+ server-side and client-side integrations, including multi-cloud and multi-CDN setups, to keep our customers safe and secure from the start.

Protect AI Apps With DataDome

DataDome has led the charge as generative AI tools have grown, from considering the potential benefits and drawbacks of allowing ChatGPT to access your content to building our own ChatGPT Companion tool to enable the creation of custom blocking rules to protect your business. We continue to gather signals from attacks every day, keeping our protection updated to catch the latest and greatest threats to our customers around the world. Generative AI tools need protection from sophisticated automated attacks—and DataDome offers best-in-class protection with unparalleled accuracy.

For a quick look at the simple bots your site is unprotected against, try our BotTester tool. You can get a more in-depth look at how DataDome can protect your site against even the most sophisticated threats when you book a demo or start a free trial.