Security

At a glanceSecurity

Every defence sits before, around, or after the model — never inside the prompt. An attacker who reaches the model has already beaten anything you wrote there.

LandscapeTypes & approaches

Click a highlighted type to open its own page — concept, use case, and diagram.

The attacker gets a vote

Guardrails keep a cooperative system inside its lane. Security is what you need when someone is actively trying to push it out. Different threat model, different defences, and almost none of the intuition transfers.

Classical software has a boundary you can point at. Code is code, data is data, and the CPU never confuses one for the other — that separation is enforced by the machine, and every injection bug in the history of computing is a story about somewhere that separation quietly broke down (SQL injection, XSS, buffer overflows). We spent fifty years learning to defend that boundary.

An LLM has no such boundary. The system prompt, the user's message, the retrieved document, the tool result, and the web page the agent just fetched all arrive as one flat sequence of tokens. There is no bit that marks "this part is instructions" and "this part is only data to be read". The model infers the difference from context, statistically, and any statistical inference can be pushed. Greshake et al. put it exactly: LLM-integrated applications "blur the line between data and instructions" [3]. That single sentence generates the entire attack surface below.

This matters more the more capable your system is. A chatbot that can only talk has a reputational problem. A RAG system has a data problem. An agent with tools, credentials and network access has a breach problem — because the model is the thing deciding which tools to call, and the model is the thing the attacker just took over.

The mapOWASP Top 10 for LLM Applications (2025)

OWASP's GenAI Security Project maintains the reference list of LLM risks, refreshed for 2025 [1]. It is the closest thing the field has to a shared vocabulary, and it is what an auditor or an enterprise security review will ask you about by ID. Learn the IDs. The right-hand column is where each one is handled on this site.

Source: OWASP Top 10 for LLM Applications, 2025 revision [1]. The 2025 list added System Prompt Leakage and Vector/Embedding Weaknesses and renamed several 2023 entries — cite the year when you reference it, because the IDs moved.

LLM01 · the root bugPrompt injection: direct and indirect

Direct injection is the one everybody pictures: a user types something that overrides your system prompt. It is real, and it is the less interesting half. The user attacking their own session is mostly attacking their own privileges — annoying, cheap, usually contained.

Indirect injection is the one that should change your architecture. The malicious instruction is not typed by the user at all. It is already sitting in content your system retrieves and drops into the context window: a web page, a PDF, a support ticket, a calendar invite, a code comment, a tool response, an email. Greshake et al. demonstrated this against real systems — including Bing's GPT-4 chat and code-completion engines — and showed it enables remote exploitation with no attacker access to the interface at all [3]. The attacker does not need to talk to your app. They need to write something your app will read.

Sit with the consequence. Your users are not the threat surface. Your corpus is the threat surface. Any document that anyone outside your trust boundary can influence is an instruction channel into your model. If your RAG index ingests public web pages, your model takes instructions from the internet. If it ingests inbound support email, your model takes instructions from anyone who knows your support address.

And it persists. An injected instruction that lands in conversation memory, a summary, a scratchpad file, or a re-indexed note becomes a stored injection — the LLM equivalent of stored XSS, firing on every future session that reads that record.

LLM04 + LLM08RAG poisoning: five documents against millions

If you build on this site's material you will build RAG. So this is the attack to internalise.

PoisonedRAG (USENIX Security 2025) is the load-bearing result. The attacker injects a small number of crafted texts into a knowledge base to force a chosen answer for a chosen question. The number required is the part that should stop you: a 90% attack success rate from injecting five malicious texts per target question, into a database containing millions of texts [4].

That is the mechanism, and it is worth being precise about it because it inverts the intuition everyone brings. Retrieval is adversarially helpful: it is optimised to surface the passage that best matches the query. An attacker who knows the target question can craft a chunk that is a near-perfect semantic match and therefore gets promoted to the top of the results, past thousands of legitimate documents. The retriever is doing its job correctly. That is the problem.

The PoisonedRAG authors also evaluated defences and found them insufficient [4] — a finding that rhymes with everything else in this section.

Alignment under attackJailbreaks: why the model itself cannot be the last line

A jailbreak targets the model's safety training rather than your application logic. For a builder, the point of studying jailbreaks is not the harmful-content angle — that is Guardrails territory. The point is what jailbreaks prove about model-level defences in general, which is what your injection defences also are.

Two results are worth knowing exactly.

Generalise that: a defence inside the model competes with the model's own capabilities and tends to lose; a defence outside the model does not have to. That principle drives the architecture section below.

Model-layer defences are still worth buying — they are just not sufficient. Anthropic's Constitutional Classifiers, an external filtering layer, cut universal-jailbreak success from 86% to 4.4%; across ~3,000 hours of red-teaming by 185 participants, no participant found a universal jailbreak covering all ten target queries. The first generation cost 23.7% additional compute [7]. That is the real shape of the trade: a large, measurable risk reduction, a real bill, and not zero.

LLM02 + LLM05Data exfiltration through tool use

Injection alone changes what the model says. Injection plus an outbound channel changes what the attacker has. The channel is rarely a tool called send_data_to_attacker. It is almost always something innocuous.

CVE-2025-32711 ("EchoLeak"), Microsoft 365 Copilot, 2025 — described in the published analysis as the first known case of prompt injection weaponised for concrete data exfiltration in a production AI system, and genuinely zero-click [20]. Read the chain as a defender, because every link is a control that exists in your system too:

Six controls. Each individually reasonable. The chain went through all of them, and the published analysis concludes that only a layered, defence-in-depth approach can contain this class of threat, because the underlying issue is the collapse of the boundary between content and commands [20].

The lethal trifecta [21] is the compression of all of this into something you can hold in your head and say out loud in a design review. An agent is exposed when it combines: (1) access to private data, (2) exposure to untrusted content, and (3) the ability to communicate externally. All three together, and an attacker can trick it into reading your data and sending it out. Remove any one, and the attack loses its shape — which is a design instruction, not an observation. Most agents acquire all three by accident, one useful feature at a time, and nobody notices because no single pull request adds more than one.

LLM06 · the structural flawThe confused deputy in agent design

This one has a name and a literature because we solved it before — in 1988. Norm Hardy described the confused deputy: a program tricked by a less-privileged party into misusing its own authority. His example was a compiler with permission to write to a system directory; a user who asked it to write debug output to the billing file got the billing file destroyed, because the compiler used its permissions, not the user's. The subtitle of that paper is the punchline: "or why capabilities might have been invented" [22].

Your agent is the compiler. It holds credentials — a database connection, an API key, an OAuth token, a service account. It receives instructions from a channel an attacker can write to (see: your corpus). It acts with its own ambient authority. That is the 1988 bug, reissued with a chat interface, and it is why "the agent has admin so it can help anyone" is a sentence that should stop a design review.

Hardy's diagnosis transfers exactly: the vulnerability comes from ambient authority — permission that applies implicitly whenever the deputy acts, decoupled from the request that triggered it. The fix transfers too. Capability systems bundle the designation of an object together with the permission to access it [22], so authority travels with the request instead of hanging in the air.

LLM03 + LLM04Supply chain: models, datasets, packages

Software supply chain security assumed artifacts are inert until executed. A model file is not inert. Serialised model formats can execute code on load, before any human or eval sees a single token of output.

LLM02 + LLM07PII leakage and memorisation

Sensitive data reaches a user through three distinct doors. They have three distinct fixes and teams routinely fix one and claim all three.

On memorisation, the strongest published result is Nasr et al.'s divergence attack: a technique that pushes a production model off its chatbot-style generation and causes it to emit training data at ~150× the normal rate. The authors extracted gigabytes of training data across open, semi-open and closed models, including ChatGPT [19]. Memorisation is not folklore, it is measured, and "the model would never say that" is not an argument.

The honest tableWhat works, what only looks like it works

This is the section to read twice. The AI security market sells a lot of things that demonstrate beautifully and fail against an attacker who knows they are there.

Start with the one everybody tries first. "Just tell the model to ignore instructions in retrieved content." It is free, it is one line, it demos perfectly, and it does not work. Three reasons, and they compound:

The empirical version of that argument is the most important defensive finding of 2025. Google DeepMind evaluated in-context defences (spotlighting, paraphrasing, in-context learning, self-reflection, perplexity filtering) and classifier defences against adaptive attackers — attackers who optimise against the deployed defence rather than against an undefended baseline. Their finding, verbatim: "Many defenses that perform well on our static evaluation set can be tricked by small and subtle adaptations to an attack." Their conclusion: "Robustness requires defense in depth. Adversarially training improves resilience to known attacks, however enumerating the space of all possible attacks is intractable" — and defences must sit at every layer of the stack [15].

Numbers are from the cited papers under their own evaluation conditions and do not transfer to your system unchanged. Read a defence's threat model before you read its percentage — and check whether the attacker was allowed to adapt.

The way outDesign patterns with provable properties

Every model-layer defence above is probabilistic: it lowers the odds. The research consensus is that you get real guarantees only by changing the architecture so that a successful injection cannot reach anything that matters.

The organising principle, from Beurer-Kellner et al. — and this is the sentence to remember from this entire page:

Note the word impossible, not unlikely. The paper starts from the assumption that the model is already compromised — an agent may do anything after ingesting untrusted input — and asks what structure survives that assumption. Their six patterns, each trading some generality for containment [13]:

CaMeL is the most complete instantiation and the one to study. It builds a protective system layer around the LLM that holds "even when underlying models are susceptible to attacks". It extracts control and data flow from the trusted query, so untrusted data can influence values but never the program's structure, and uses capabilities — the direct descendant of Hardy's 1988 answer — to block unauthorised exfiltration through tool calls [12].

And it publishes its price honestly, which is why it is worth citing in a design review: on AgentDojo, CaMeL solved 77% of tasks with provable security, versus 84% for an undefended system [12]. Seven points of capability for a security property you can actually reason about. That is the trade, stated plainly. Most vendors will not state theirs.

Measure your own system: AgentDojo is the open benchmark for exactly this — 97 realistic tasks across email, banking and travel, with 629 security test cases, built as a dynamic environment so you can add adaptive attacks rather than replay a fixed corpus [14]. Its baseline finding is a useful humility check: current LLMs fail many of these tasks even with no attacker present.

In practiceWhat to actually do

Ordered by leverage. If you do the first three and nothing else, you have eliminated most of what this page describes.

Questions you will be askedStraight answers

ReferencesSources

Every external claim on this page traces to one of these. Security moves fast — the papers below are dated on purpose. Re-check the OWASP list and the agent-defence literature at least twice a year; a defence that was state of the art in 2024 may be a known-broken one now.