A reliable retrieval-augmented generation (RAG) app is less about connecting a language model to a vector database and more about controlling the full path from source document to final answer. This guide explains a practical RAG architecture, shows how to estimate retrieval and generation costs with your own inputs, and provides an evaluation and production checklist you can revisit as models, traffic, and document collections change.
Overview
RAG combines an information-retrieval step with an LLM response step. When a user asks a question, the application converts the question into an embedding, searches an indexed document collection, selects relevant passages, and places those passages in the model's context. The model then generates an answer grounded, ideally, in that retrieved material.
A production RAG system usually contains six layers:
- Ingestion: Collect documents and preserve useful metadata such as title, source, date, access level, and section.
- Parsing and chunking: Convert files or pages into clean text and divide it into retrievable units.
- Embedding: Represent documents and queries as vectors so semantically related content can be found.
- Retrieval: Search a vector database, keyword index, or hybrid system and rank candidate passages.
- Generation: Give the selected context to an LLM with instructions about evidence, uncertainty, and format.
- Evaluation and monitoring: Test retrieval and answer quality separately, then observe cost, latency, failures, and user feedback.
The most common design mistake is evaluating only the final prose. A fluent answer can still be based on irrelevant or incomplete passages. Test whether the right evidence was retrieved before deciding whether the prompt or model is responsible for the result. For related guidance, see this prompt testing framework and the practical guide to reducing LLM hallucinations in production.
How to estimate
Estimate a RAG app using one representative request rather than a vague monthly guess. Then multiply that request profile by expected traffic. Keep retrieval, generation, storage, and operational overhead as separate line items so a change in one layer does not hide the others.
A simple request-cost model is:
Cost per request = query processing cost + retrieval cost + generation cost + application overhead
For token-based model pricing, use:
Generation cost = (input tokens ÷ 1,000,000 × input rate) + (output tokens ÷ 1,000,000 × output rate)
Apply the same calculation to embedding requests if your provider charges for embedding input. Your monthly estimate can then be expressed as:
Monthly variable cost = cost per request × monthly requests
Add fixed or usage-based infrastructure separately, including vector storage, database operations, document-processing jobs, logging, monitoring, and any reranking component. If the system has multiple request types, calculate each type independently:
Total monthly estimate = (research requests × research cost) + (short-answer requests × short-answer cost) + infrastructure
This approach is useful even when exact provider rates are not yet selected. Enter your current rates when you have them, keep assumptions in a spreadsheet or configuration file, and recalculate when those inputs change. The AI app cost breakdown can help you organize the broader cost categories.
Inputs and assumptions
Record these inputs before choosing a model or database. Clear assumptions make technical decisions easier to compare.
Traffic and request shape
- Expected requests per day or month
- Percentage of questions that require retrieval
- Average query length in tokens
- Number of retrieved chunks placed in context
- Average tokens per chunk, including metadata
- Expected answer length
- Peak concurrency and acceptable response time
Do not assume every request has the same shape. A short factual lookup and a multi-document comparison may use different retrieval limits, prompts, and output lengths.
Document and retrieval assumptions
- Number of source documents and their approximate size
- How often documents are added, deleted, or re-indexed
- Chunk size and overlap
- Whether headings, tables, URLs, dates, and permissions are preserved
- Top-k retrieval value and any reranking step
- Whether keyword, vector, or hybrid search is most appropriate
Chunking is a quality decision as well as a cost decision. Very small chunks can lose context and increase the number of passages needed. Very large chunks may dilute relevance and consume more context. Start with a documented rule based on the structure of your material, then compare alternatives with a fixed evaluation set instead of changing several variables at once.
Reliability assumptions
Define what the app should do when retrieval is weak. A useful system prompt should tell the model to answer from supplied evidence, distinguish evidence from inference, cite the source or section when possible, and say that it cannot verify an answer when the context is insufficient. Add a confidence or evidence threshold in application logic rather than relying only on wording in the prompt.
Also decide how the system handles access control, stale documents, duplicate passages, prompt injection inside retrieved text, provider timeouts, and malformed model output. A structured-output response can make citations and refusal states easier to validate before displaying them.
Worked examples
Example 1: Estimating a single request
Assume a request contains 250 query tokens, retrieves five chunks of 350 tokens each, adds 200 tokens of instructions and metadata, and produces a 180-token answer. The estimated model input is:
250 + (5 × 350) + 200 = 2,200 input tokens
The output is 180 tokens, so the request profile is 2,200 input tokens and 180 output tokens. Insert the current input and output rates for your selected model into the token formula. If the application performs a separate query-embedding call, calculate that call independently using the query length and embedding rate.
Now compare a lower-context design with three chunks against the five-chunk design. The lower-context version may reduce input usage, but it is only an improvement if retrieval quality and answer completeness remain acceptable. Record both cost and evaluation results rather than optimizing tokens in isolation.
Example 2: Comparing request classes
Suppose a publisher expects 8,000 short-answer requests and 2,000 research requests in a month. Short answers use three chunks and a short response; research requests use eight chunks, a reranking step, and a longer response. Calculate each request class separately, then add the fixed database and monitoring costs. This makes the estimate more useful than multiplying one average request by 10,000, because a change in the mix of request types will be visible.
Example 3: Measuring quality
Create a small, representative evaluation set containing questions, expected source documents, acceptable answer points, and known unanswerable questions. Track at least four outcomes: whether a relevant passage was retrieved, whether the answer was supported by that passage, whether citations pointed to the correct source, and whether the app abstained when evidence was missing. The guide to creating evaluation datasets provides a useful companion workflow.
When to recalculate
Recalculate the estimate whenever a pricing input, benchmark, or traffic assumption moves. In practice, review it after changing the embedding model, generation model, chunking rule, retrieval depth, reranker, prompt, output limit, vector database, or logging level. Review it as well when the document collection grows, re-indexing becomes more frequent, or users begin asking more complex questions.
Use a lightweight change log with four fields: date, changed input, old assumption, and new result. For each release, run the same evaluation set and compare cost, latency, retrieval quality, citation accuracy, and abstention behavior. Observability becomes especially valuable when production traffic differs from test data; consider tracking traces, token usage, retrieval results, errors, and user feedback together. See the comparison of LLM observability tools for the categories worth monitoring.
Before launch, confirm that you can:
- Rebuild the index from source documents.
- Trace an answer back to retrieved passages and document versions.
- Reject unauthorized or stale content.
- Return a useful fallback when evidence is missing.
- Measure retrieval and answer quality independently.
- Recalculate cost from stored request inputs rather than guesses.
Start with a narrow document set and a fixed evaluation dataset. Establish a baseline, ship only the retrieval and answer behavior you can observe, and then expand coverage. That sequence turns RAG from a one-time demo into maintainable AI app development: measurable, debuggable, and ready to be recalculated as its inputs change.
