RAG Tutorial: Build a Reliable Retrieval-Augmented Generation App

Overview

Retrieval-augmented generation, or RAG, combines information retrieval with text generation. Instead of asking a language model to answer entirely from its trained knowledge, your application first searches a controlled collection of documents. It then supplies the most relevant passages to the model as context for the answer.

A typical RAG app has six stages:

1. Ingestion: Collect documents and record their source, owner, date, access rules, and version.
2. Preparation: Extract text, remove irrelevant content, preserve useful structure, and divide documents into chunks.
3. Embedding: Convert each chunk into a numerical representation that supports semantic search.
4. Indexing: Store embeddings, text, and metadata in a vector-capable search system.
5. Retrieval: Search for relevant chunks, optionally using keyword search, filters, reranking, or hybrid retrieval.
6. Generation: Provide selected context to the LLM and require an answer that is grounded in that context.

The goal is not to retrieve as much text as possible. It is to retrieve enough relevant, understandable evidence for the model to answer the user’s question without introducing unnecessary noise. If you are planning a broader implementation, use this guide alongside the production checklist for reliable RAG apps.

Checklist by scenario

1. Before ingesting documents

• Define the questions the application must answer. A support assistant, internal policy search tool, and research interface may require different retrieval strategies.
• List authoritative sources and identify which sources are outdated, duplicated, incomplete, or restricted.
• Assign stable identifiers to documents and retain source URLs, titles, publication dates, categories, and version information.
• Decide how deleted, replaced, or access-controlled documents will be removed from the index.
• Separate content that can be shown to users from internal metadata or confidential fields.

Do not treat ingestion as a one-time upload. A RAG system inherits the quality and freshness of its source collection, so document ownership and update procedures should be defined before launch.

2. When extracting and chunking text

• Preserve headings, lists, table labels, code blocks, and other structure that affects meaning.
• Remove navigation menus, repeated headers, cookie notices, and decorative text from web pages where practical.
• Use chunks that are small enough to retrieve precisely but large enough to preserve the surrounding explanation.
• Test overlap rather than assuming that more overlap always improves results.
• Store the original document ID and section path with every chunk.
• Keep a way to reconstruct the source passage around a retrieved chunk for citations and debugging.

There is no universal chunk size that works for every corpus. Product documentation, legal clauses, transcripts, and source code have different boundaries. Start with natural sections, then adjust after reviewing retrieval examples.

3. When designing embeddings and search

• Use the same embedding approach consistently for indexed content and incoming queries unless your system explicitly supports a compatible alternative.
• Choose metadata fields that answer practical filtering questions, such as product, language, audience, region, document type, or effective date.
• Use metadata filters when relevance alone is insufficient. For example, a user may need current documentation for one product rather than similar material from every product.
• Consider hybrid retrieval when exact terms, identifiers, or error messages matter alongside semantic similarity.
• Inspect retrieved passages directly instead of judging search quality only from the final answer.

A vector database is an implementation component, not a substitute for retrieval design. Compare systems based on the features your application needs, including filtering, indexing workflow, backups, observability, and operational complexity. The vector database comparison guide can help frame that decision without assuming one tool fits every project.

4. When writing the generation prompt

• Tell the model what the retrieved context represents and where it appears in the request.
• Require it to distinguish supported claims from missing information.
• Instruct it not to invent an answer when the context does not support one.
• Define the expected citation format and connect citations to source identifiers.
• Specify how to handle conflicting documents, unclear questions, and requests outside the system’s knowledge scope.
• Use structured output when downstream code needs fields such as answer, citations, confidence indicators, or follow-up questions.

A useful system instruction might be: “Answer using only the supplied sources. If the sources do not establish an answer, say what is missing. Cite the source IDs for each material claim.” The exact wording should be tested against representative questions rather than treated as a guarantee.

5. When adding citations and user controls

• Show citations that lead to the relevant document or passage, not merely a generic source list.
• Expose enough context for a user to verify an important claim.
• Allow users to refine searches by product, date, document type, or other meaningful metadata.
• Provide a clear fallback when retrieval returns weak or contradictory evidence.
• Log the query, filters, retrieved chunk IDs, model request, response, and latency according to your data-handling requirements.

What to double-check

Before calling a RAG app production-ready, review the complete path from question to answer.

Retrieval quality

Create a small evaluation set containing real user questions, expected source documents, and notes about acceptable answers. Check whether the correct evidence appears in the retrieved results, even when the generated response is excluded from the test. This separates search problems from prompt or model problems.

Answer grounding

Review whether each significant claim is supported by the retrieved context. Test questions that have no answer in the corpus, contain ambiguous terms, or combine information from multiple documents. A reliable application should be able to acknowledge missing evidence instead of filling the gap with plausible text.

Freshness and access

Confirm that updates are reflected in the index and that obsolete versions are not returned alongside current ones. Test access controls with both allowed and disallowed users. Filtering after retrieval may be too late if restricted text has already entered the model context, so access rules should be considered during retrieval.

Cost and performance

Measure ingestion time, embedding work, search latency, model latency, context length, and retry behavior. Track these separately so that an optimization targets the actual bottleneck. For a broader budgeting framework, see the guide to tokens, retrieval, hosting, and hidden AI app costs.

Evaluation workflow

Run a fixed test set whenever you change chunking, embeddings, filters, retrieval count, reranking, prompts, or models. Record failures by category: missed evidence, irrelevant evidence, unsupported claim, citation error, refusal error, formatting error, or latency issue. A prompt testing workflow can be extended to cover the retrieval layer as well.

Common mistakes

• Indexing everything without cleaning it: Duplicate and low-value text can crowd out the passages that matter.
• Using arbitrary chunks: Splitting in the middle of a procedure, table, or policy clause can remove essential context.
• Returning too many results: A larger context is not automatically better. Irrelevant passages can make answers less focused.
• Ignoring metadata: Semantic similarity cannot reliably enforce date, product, language, or permission requirements by itself.
• Testing only successful questions: No-answer, ambiguous, adversarial, and outdated-document cases reveal important failure modes.
• Trusting citations automatically: A citation marker is useful only when it points to evidence that supports the associated claim.
• Changing several variables at once: If chunking, retrieval count, prompt wording, and model all change together, it becomes difficult to identify what improved or harmed results.
• Skipping deletion workflows: A document refresh process must handle removed and superseded content, not just new uploads.

Keep a small set of difficult examples in version control. These examples become a practical regression suite for your LLM app development process and make improvements easier to verify.

When to revisit

Revisit this checklist before seasonal planning cycles, major product releases, documentation migrations, or any workflow change that alters the underlying knowledge sources. Also review it whenever you change the embedding model, vector database, retrieval logic, reranker, generation model, system prompt, citation format, or access-control design.

At a minimum, schedule a recurring knowledge-base review. Confirm that source owners still exist, documents are current, deleted pages are removed, metadata remains consistent, and evaluation questions still represent real usage. Compare recent failures with your original test set; new user behavior often exposes gaps that were not visible during development.

Use this final action list when preparing an update:

1. Export a sample of recent queries and anonymize sensitive content where necessary.
2. Label retrieval and answer failures separately.
3. Inspect source freshness, duplicates, permissions, and metadata coverage.
4. Change one retrieval or generation variable at a time.
5. Run the regression set and review citations manually for high-impact use cases.
6. Deploy gradually, monitor traces and costs, and document the change.

A dependable RAG application is maintained as a system, not perfected through a single prompt. Keeping the corpus, retrieval layer, generation instructions, evaluation set, and operational logs aligned gives you a repeatable way to improve quality as the product and its knowledge sources change.