Prompt Injection Defense Checklist for LLM Apps and AI Workflows

Overview

A prompt injection occurs when text supplied to an LLM influences it to ignore, reinterpret, or override the application’s intended instructions. The text may be written directly by a user, or it may arrive indirectly through a webpage, document, email, support ticket, database record, or retrieved passage. In a RAG system, for example, a retrieved document can contain instructions aimed at the model rather than information relevant to the user’s question. In an agent, the consequences can be broader because the model may have access to tools, external systems, or private context.

The central design principle is simple: treat model input as data, not authority. Delimit untrusted content, state its role clearly, and enforce important security decisions outside the model. A system prompt can explain boundaries, but it should not be the only control protecting credentials, customer records, publishing systems, payment actions, or other consequential resources.

Use the checklist below during design reviews, before deployment, and whenever a workflow gains a new tool or data source. For background on permissions and tool access, see the AI Agent Security Checklist. If your application relies on structured responses, pair this review with guidance on structured output validation.

A threat-to-control view

Checklist by scenario

1. User-facing chat and content generation

• Define the application’s permitted task in plain language and keep it narrow enough to test.
• Separate system instructions, user input, retrieved context, and tool results in the request format. Clear labels help, but do not treat labels as a security boundary.
• Assume users will ask for hidden prompts, internal notes, credentials, or unrelated operations. Decide what the application should disclose and test refusal behavior.
• Remove unnecessary secrets and personal data from the model context. A model cannot reliably protect information that the application unnecessarily exposes.
• Validate generated content before displaying it as HTML, executing code, publishing it, or sending it to another person.
• For content operations, require factual review for claims that matter. The AI content fact-checking workflow provides a useful editorial pattern for claim extraction and sign-off.

2. RAG applications and document search

• Classify every data source by trust level, ownership, sensitivity, and intended use.
• Store document text separately from application instructions. Retrieved passages should be quoted as evidence or context, not merged into the governing prompt as if they were commands.
• Instruct the model to use retrieved material only to answer the defined question. It should not follow operational instructions found inside a document unless a separate, authorized process approves them.
• Filter retrieval results by user permissions before they reach the model. Do not rely on the model to decide whether a user is allowed to see a document.
• Log document identifiers, query details, and retrieval decisions without storing unnecessary sensitive content.
• Test poisoned documents, misleading metadata, irrelevant instructions, and attempts to make the model cite or reveal unrelated passages.

Context size also affects reliability: excessive retrieved text makes it harder to inspect what the model received and can increase the number of places where hostile instructions appear. Use the budgeting techniques in this LLM token and context window guide.

3. Tool-using agents and API workflows

• Create an explicit allowlist of tools. If a capability is not required for the task, do not expose it.
• Give each tool the minimum permissions and narrowest data access needed. Separate read operations from write, delete, publish, or send operations.
• Enforce authorization in application code on every tool call. The model’s decision to call a tool is not proof that the caller is authorized.
• Validate all arguments against a strict schema, including identifiers, URLs, file paths, quantities, recipients, and filters.
• Block requests to internal network locations, arbitrary file paths, shell commands, or unrestricted URLs unless the use case specifically requires them and has additional safeguards.
• Require confirmation for irreversible, external, financial, privacy-sensitive, or public actions. Show the user the exact action and parameters being approved.
• Set timeouts, rate limits, retry limits, and spending or volume caps. Review the related LLM API error-handling patterns so failures do not become repeated unsafe actions.

4. Automated publishing and AI SEO workflows

• Keep research, drafting, fact checking, formatting, and publishing as separate stages with distinct permissions.
• Do not let a model-promoted instruction inside a source document change publishing rules, target URLs, canonical settings, or editorial requirements.
• Validate links, metadata, markup, and output format before publication.
• Use a human approval gate for pages involving sensitive claims, regulated topics, reputational risk, or large-scale changes.
• Retain an audit record showing the input sources, prompt or configuration version, model output, validation results, and approving person or system.

5. Structured output and code generation

• Use a schema with required fields, allowed values, length limits, and sensible numeric ranges.
• Parse and validate the response before downstream code uses it. A response that looks like JSON is not necessarily valid or safe JSON.
• Escape output for its destination, such as HTML, SQL, shell, Markdown, or a browser context. Schema validation does not replace context-specific sanitization.
• Reject or quarantine invalid responses rather than silently coercing them into an action.
• For generated code, use isolated execution, restricted permissions, dependency controls, and human review. Never execute model output merely because it passed a formatting check.

What to double-check

Before approving an LLM workflow, trace the complete path from input to consequence. Identify every place where text enters the system, every place where the model can see private information, and every place where its output can affect another system.

1. Trust boundaries: Can the team distinguish instructions from user content, retrieved content, tool results, and memory? Are those categories represented separately in logs and tests?
2. Authority: Which decisions are made by deterministic application code rather than the model? Are permissions checked again at the point of action?
3. Secrets: Are credentials kept out of prompts, conversation history, tool descriptions, and error messages? Are logs reviewed for accidental disclosure?
4. Output handling: What happens when the model returns extra fields, malformed data, a refusal, an unexpected URL, or a request to call a different tool?
5. Human approval: Is approval specific and informed, or is a person merely asked to click through a vague confirmation? The reviewer should see the proposed action, target, scope, and relevant source.
6. Monitoring: Can the team detect unusual tool sequences, repeated refusals, high-volume extraction, unexpected destinations, or attempts to access restricted data?
7. Testing: Do evaluations include direct attacks, indirect attacks, multilingual inputs, long-context distractions, encoded instructions, misleading documents, and benign requests that should still work?

Keep attack tests alongside ordinary quality tests. A prompt change that improves helpfulness may also change refusal behavior, tool selection, or data exposure. A repeatable prompt testing framework can help compare these behaviors across model, prompt, and workflow changes.

Reusable security review template

Record the following for each workflow:

• Workflow and owner: Name, purpose, business owner, and technical owner.
• Inputs: User fields, files, retrieval sources, memory, web content, and tool results.
• Model authority: What the model may decide, and what it may never decide.
• Tools and permissions: Tool list, read/write scope, parameter constraints, rate limits, and authorization checks.
• Sensitive data: Data categories exposed, masking rules, retention, and access controls.
• Controls: Isolation, validation, sanitization, monitoring, and human approval points.
• Test evidence: Attack cases, expected behavior, observed behavior, unresolved findings, and retest date.
• Release decision: Approved, approved with conditions, or blocked, with an accountable reviewer.

Common mistakes

• Relying on a longer system prompt: More instructions do not compensate for excessive permissions or missing server-side checks.
• Calling delimiters a defense: Delimiters improve clarity, but an attacker-controlled passage remains untrusted content.
• Giving agents broad access for convenience: Convenience turns a model error into a larger application incident. Start with the smallest useful capability.
• Validating only the final prose: Inspect tool calls, arguments, retrieved sources, intermediate data, and side effects as well.
• Treating refusal as proof of safety: A model may refuse one visible attack while still leaking information through a tool call, citation, error message, or later turn.
• Skipping ordinary security controls: Authentication, authorization, network restrictions, secret management, logging, and rate limits remain necessary around an LLM.
• Approving large batches at once: Use sampling, staged rollouts, dry runs, and rollback plans for automated content or data changes.

When to revisit

Review this checklist before each production deployment and whenever the workflow changes. A new model, prompt, retrieval index, connector, tool, memory feature, user group, or publishing destination can change the threat profile even when the visible product experience appears unchanged.

Schedule a recurring review before seasonal planning cycles and other periods when traffic, content volume, or business rules change. Revisit immediately after a red-team finding, unexpected model behavior, data exposure, failed authorization check, or provider-side change that affects model behavior or safety controls. Record the date, scope, test cases, findings, and owner for every review so the next reviewer can compare results rather than starting from memory.

For a practical final pass, ask one person to act as an attacker and another to trace controls from input to side effect. Run direct and indirect injection tests, inspect tool permissions, verify output schemas, review logs for sensitive data, and confirm that high-impact actions stop for human approval. Then retest the ordinary user journey to ensure the defenses preserve the task the application is meant to perform. Prompt injection prevention is not a one-time prompt engineering exercise; it is an ongoing model reliability practice built from constrained authority, observable behavior, and repeatable evaluation.