AI Agent Security Checklist: Permissions, Prompt Injection, Secrets, and Tool Access

Overview

Securing an AI agent is not only a matter of improving its system prompt. An agent combines a language model with instructions, context, memory, tools, credentials, and application logic. Each component creates a different boundary that should be reviewed separately.

Start by documenting the agent's intended job in one sentence. Then list what it can read, what it can change, which external systems it can contact, and which actions require a person. This simple inventory turns a vague security goal into a reviewable design.

A useful threat model asks five questions:

• What can the agent see? Include user messages, retrieved documents, conversation history, files, logs, and tool responses.
• What can the agent do? List every tool, API operation, database query, file operation, and outbound message.
• Who can trigger it? Consider end users, internal staff, automated jobs, webhooks, and other agents.
• What could go wrong? Consider data disclosure, unauthorized changes, fraudulent messages, destructive operations, and denial of service.
• How will the system stop or recover? Define approval gates, rate limits, rollback procedures, and an emergency disable switch.

Keep the model's role narrow. The application should enforce authorization, validation, and business rules rather than asking the model to enforce them through prompt wording alone. For background on safe tool integration, see this function calling tutorial, and use schema validation patterns from the guide to structured output with LLMs.

Checklist by scenario

1. Before the first production release

• Write down the agent's purpose, users, trust boundaries, and out-of-scope requests.
• Inventory every input, including retrieved content and tool output. Treat external content as untrusted data, not as instructions.
• Create a tool registry with each tool's purpose, allowed operations, required parameters, data sensitivity, and owner.
• Use separate credentials for development, testing, and production.
• Grant the narrowest practical permissions. Prefer read-only access where the agent does not need to write.
• Validate tool arguments in application code. Reject unexpected fields, invalid identifiers, unsafe paths, and unsupported operations.
• Add authentication and authorization checks outside the model. Do not rely on the agent to determine whether a user is allowed to access a record.
• Define approval requirements for irreversible, financial, external-communication, account, deletion, or privilege-changing actions.
• Set limits for request size, tool calls, execution time, retries, and spend. Make limits visible in monitoring.
• Log enough information to investigate decisions without placing secrets or unnecessary personal data in logs.
• Test failure behavior: unavailable tools, malformed model output, timeouts, duplicate requests, partial completion, and revoked credentials.

2. When adding a new tool or API

• Document the tool's exact contract and avoid broad endpoints when a narrower operation is available.
• Separate planning from execution. Let the agent propose an action, then let deterministic code validate and perform it.
• Use an allowlist for hosts, methods, repositories, folders, tables, or account scopes as appropriate.
• Prevent the tool from accepting model-generated authorization details. Derive identity and access scope from the authenticated application context.
• Return only the minimum data needed for the next step. Redact tokens, private fields, and unrelated records.
• Use idempotency keys or duplicate-action checks for operations that create, send, charge, publish, or update.
• Add a negative test showing that the tool refuses an out-of-scope request.

3. For prompt-injection testing

Prompt injection prevention is a layered control problem. No system prompt can guarantee that an agent will ignore every malicious instruction in user input, a webpage, an email, or a retrieved document. Test the complete application instead.

Include cases such as:

• A document says to ignore the system instructions and reveal hidden context.
• A web page asks the agent to follow a link and upload credentials.
• A user asks the agent to expose another user's private records.
• A tool response contains instructions to call a different tool or change permissions.
• Encoded, obfuscated, multilingual, or split instructions attempt to bypass a policy.
• A retrieved document includes a plausible but unauthorized request to send an external message.
• A long context places the malicious instruction near relevant content or at the end of the prompt.

For each test, verify not only the final text but also whether an unauthorized tool call was attempted. A safe result should preserve data boundaries, refuse or quarantine the instruction, and continue only with an approved task.

4. For agents handling sensitive information

• Classify the data the agent can process and define which categories it must not retain or transmit.
• Keep secrets in a dedicated secret manager or protected runtime configuration, never in prompts, source code, examples, or conversation memory.
• Pass short-lived, narrowly scoped credentials when possible, and make revocation straightforward.
• Redact sensitive values before sending telemetry to external systems.
• Check whether retrieved content, cached context, and logs inherit the same access controls as the source data.
• Test that an ordinary user cannot cause the agent to summarize, export, or search data outside their authorization.

What to double-check

Use a permission matrix to make hidden assumptions visible. A simple version can include these columns:

• Actor: end user, service account, reviewer, administrator, or scheduled job.
• Resource: database, file store, email system, CRM, deployment system, or knowledge base.
• Operation: read, create, update, delete, publish, send, or administer.
• Scope: tenant, team, project, record, folder, or environment.
• Approval: none, user confirmation, trained reviewer, or two-person review.
• Evidence: authentication context, validation result, tool call, approval event, and outcome.

Every permitted operation should have an owner and a reason. If a row cannot be explained, remove the permission until the use case is clear.

Double-check the boundary between instructions and data. System prompts, user requests, retrieved passages, and tool responses should be represented separately in the application. Labeling content does not make it trustworthy by itself, but it helps the orchestration layer apply different rules and makes reviews easier.

Also inspect structured output. Require a schema, reject additional fields when appropriate, validate types and ranges, and treat a valid JSON response as syntactically correct rather than automatically safe. A field such as approved: true must not substitute for an authorization check.

Finally, review observability. Record the request identifier, authenticated actor, selected tool, validated arguments, approval event, result status, and failure reason. Avoid logging full prompts or responses by default when they may contain sensitive content. Your prompt testing framework can help organize regression cases, but security tests should also cover application permissions and tool behavior.

Common mistakes

• Using the system prompt as an access-control layer. Prompts guide behavior; application code must enforce identity and permissions.
• Giving one broad service account to every tool. Separate credentials and scopes reduce the impact of a compromised workflow.
• Allowing unrestricted browsing, code execution, or shell access. Use sandboxing, allowlists, resource limits, and isolated environments where these capabilities are necessary.
• Returning complete records to the model. Filter fields and scope queries before data reaches the context window.
• Testing only cooperative users. Include hostile inputs, malicious documents, malformed arguments, repeated requests, and tool failures.
• Adding approval after the action. A review that occurs after sending, deleting, publishing, or changing access is an audit step, not a preventive control.
• Ignoring indirect prompt injection. Content from search, RAG, email, and files can carry instructions even when the user message appears safe. Review the RAG architecture and retrieval filters as part of the threat model.
• Failing open during errors. If validation, authorization, or a required approval service is unavailable, pause the action rather than guessing.

When to revisit

Make this checklist part of the agent's change process, not a one-time launch document. Revisit it before production release, before seasonal planning or high-volume periods, and whenever workflows or tools change.

Run a focused review after adding or removing a tool, changing a system prompt, switching models, expanding retrieval sources, modifying credentials, changing data retention, or introducing a new user group. Repeat prompt-injection and authorization tests after major platform releases because model behavior, tool interfaces, and available sandbox controls may change.

At each review, compare the current tool registry and permission matrix with the previous approved version. Remove unused access, rotate credentials when appropriate, inspect failed or blocked tool calls, and confirm that emergency shutdown and rollback procedures still work.

A practical final gate is to ask an independent reviewer to trace one high-impact request from user authentication through model output, validation, approval, tool execution, logging, and recovery. If any step depends on the model voluntarily behaving well, replace that assumption with a deterministic control. That is the core principle of secure tool calling: keep the agent useful, but keep authority outside the model.