A prompt testing framework turns subjective LLM output review into a repeatable quality process. This guide provides a reusable test-case template, scoring model, regression workflow, and practical examples for comparing prompts, models, and application changes without relying on occasional spot checks.
Overview
Prompt engineering does not end when a response looks good in a chat window. A prompt can perform well on one example and fail when the user changes the wording, omits context, supplies unexpected data, or asks for a task outside the intended scope. In an AI application, those failures can affect search results, support answers, structured records, content drafts, or downstream tool calls.
Prompt evaluation is the practice of testing outputs against predefined expectations. The goal is not to prove that a model is universally correct. It is to determine whether a particular prompt and model combination behaves acceptably for a defined use case.
A useful evaluation workflow has five parts:
- Test cases: representative inputs that reflect real usage.
- Expected behavior: observable requirements for a satisfactory response.
- Scoring: a consistent method for judging quality.
- Regression checks: comparisons against a prior prompt or model version.
- Review and maintenance: a process for adding cases as failures appear.
This structure works for a simple summarizer, a retrieval-augmented generation system, an AI content workflow, or an agent that uses tools. It is also flexible enough to support manual review at the beginning and automated checks as the test suite matures.
Template structure
Store each evaluation case as a record rather than as an informal note. A spreadsheet, JSON file, database table, or dedicated prompt testing tool can work if the fields remain consistent. The following template captures the information needed to reproduce and judge a test.
{
"case_id": "support_001",
"category": "missing_information",
"input": "Can I change my delivery address?",
"context": "The user has not provided an order number or delivery status.",
"expected_behavior": [
"Acknowledge the request",
"Ask for the information needed to proceed",
"Do not claim that the address was changed"
],
"must_not_contain": [
"A fabricated confirmation",
"An invented order status"
],
"scoring": {
"instruction_following": 0,
"factuality": 0,
"format": 0,
"safety_or_scope": 0
},
"notes": "Use for prompt and model regression checks"
}Keep the input and context separate when your application supplies both. This makes it easier to identify whether a failure came from the user request, retrieved material, conversation history, or prompt instructions.
Define observable expectations
A weak expectation says, “Give a helpful answer.” A stronger one says, “If the required account identifier is missing, ask for it and do not imply that an account change has already occurred.” The second version can be reviewed consistently by a person or translated into a programmatic check.
Use several dimensions rather than one broad quality score. Common dimensions include:
- Instruction following: Did the output perform the requested task?
- Factuality or groundedness: Are claims supported by the supplied information?
- Completeness: Did it include the required points?
- Format validity: Does it match the required schema, headings, or field types?
- Scope control: Did it refuse, clarify, or redirect when appropriate?
- Style: Does it meet the application’s tone and length requirements?
A simple scale such as 0 to 2 is often sufficient: 0 for failure, 1 for partial compliance, and 2 for clear compliance. The scale matters less than documenting what each value means.
How to customize
Begin with the application’s highest-risk behaviors, not a large collection of easy examples. If you are building a structured-output LLM feature, test invalid inputs, missing fields, long values, unexpected characters, and cases where the correct answer is “unknown.” If you are building a RAG application, test questions that are answerable from the retrieved context, questions that are not supported, conflicting sources, and irrelevant retrieval results. A practical RAG evaluation should distinguish retrieval failure from generation failure.
Organize cases into categories that reveal patterns:
- Happy path: Clear requests with complete information.
- Ambiguous requests: Inputs that should trigger a clarifying question.
- Boundary cases: Empty, unusually long, malformed, or out-of-range inputs.
- Adversarial cases: Instructions that conflict with the application’s intended behavior.
- Unsupported requests: Questions the system cannot answer from its available information.
- Format cases: Inputs that test JSON validity, required fields, and type constraints.
Separate deterministic checks from judgment-based checks. A parser can verify that JSON is valid, required keys exist, and a number falls within an allowed range. Human or model-assisted review may still be needed for factual accuracy, relevance, and tone. If a model is used as a grader, define the rubric carefully and periodically compare its decisions with human review.
For regression testing, preserve a baseline. Run the same cases against the previous prompt and the candidate prompt, then compare both aggregate scores and individual failures. A higher average score does not automatically justify a change if the new version introduces a severe failure in a critical category.
When testing multiple models, keep other variables stable where practical: use the same cases, context, output schema, and scoring rules. Record the model identifier, prompt version, test date, relevant generation settings, and any retrieval or tool configuration. This metadata makes results interpretable when a failure reappears later.
Examples
Example 1: Content brief generation
Suppose a publishing workflow asks an LLM to turn a keyword and audience description into a content brief. A test case might require one proposed title, a defined search intent, five subtopics, and a list of questions that the article should answer. The expected behavior should also say what happens when the keyword is vague or commercially sensitive.
Useful checks include whether every required field is present, whether the subtopics match the stated intent, and whether the output invents unsupported product claims. A format validator can catch missing fields, while an editor can assess topical usefulness and duplication.
Example 2: RAG answer with insufficient evidence
Provide context that discusses a product’s setup but says nothing about its pricing. Ask, “What does this product cost?” The expected behavior is not a guessed price. The system should state that the supplied context does not establish the answer and, depending on the application, suggest where the user can verify it.
This case tests groundedness and refusal behavior together. It is especially valuable because a fluent answer can appear credible while exceeding the evidence.
Example 3: Tool or function selection
For an AI agent, create cases where a tool is clearly required, where no tool is needed, and where a required parameter is missing. Score the selected function, argument names and types, and the decision to ask a follow-up question. Include a case in which the user requests an irreversible action without confirmation if your application supports such actions.
These examples can be linked to a broader prompt engineering framework and combined with the evaluation practices in this guide to creating eval datasets.
When to update
Treat the evaluation set as a living part of the application. Update it whenever the system prompt, model, retrieval pipeline, output schema, tool definitions, or user-facing workflow changes. A change to source content can also justify new cases, particularly when the application answers questions about frequently revised material.
Add a regression case after every meaningful production failure. Record the original input, relevant context, observed output, expected behavior, severity, and the change intended to address it. Do not delete an old case merely because the prompt now passes it; historical failures are valuable evidence that the test suite is exercising real risk.
Use a practical release routine:
- Run the full evaluation set against the current version.
- Compare results with the approved baseline.
- Inspect every new failure, not only the aggregate score.
- Run focused tests for the area changed.
- Document accepted trade-offs and unresolved issues.
- Promote important production failures into permanent cases.
Start small: a carefully written set of cases is more useful than a large set with vague criteria. As the application grows, divide evaluations into fast checks for every change and deeper reviews for scheduled releases. For workflows that publish or update content, pair automated checks with human review; the human-in-the-loop workflow explains how to add that control without reviewing every low-risk output.
The durable principle is simple: define what good looks like before comparing outputs. A prompt testing framework makes that definition visible, repeatable, and easier to improve whenever the prompt, model, data, or publishing workflow changes.
