LLM token counting is the foundation of predictable AI app development. This guide shows how to estimate prompt size, reserve room for output, calculate approximate request costs, and keep retrieved documents and conversation history inside a model’s context window without relying on guesswork.
Overview
A token is a unit of text processed by a language model. It is not the same as a character, word, or byte. Short words may fit into one token, while uncommon words, code, punctuation, URLs, and non-English text can be split into several tokens. The exact result depends on the model’s tokenizer.
Every LLM request has at least two token budgets to consider:
- Input tokens: system instructions, user messages, conversation history, retrieved content, tool definitions, and formatting instructions.
- Output tokens: the response the model generates, including structured JSON, explanations, citations, or tool arguments.
The model’s context window is the maximum combined space available to the request and response. A request can fail, be truncated, or produce an incomplete answer when the input consumes nearly all available context and no output room remains. Context limits and pricing units differ by provider and model, so treat provider documentation as the source of truth whenever you select a model.
Token budgeting is therefore more than counting words. It is a repeatable planning process that connects prompt design, retrieval settings, output requirements, latency, and API cost. For broader cost and latency planning, see the LLM cost and latency optimization workflow.
How to estimate
Use this basic equation before sending a request:
Estimated request tokens = fixed instructions + conversation history + retrieved content + current user input + tool or schema overhead
Then reserve output capacity:
Required context capacity = estimated input tokens + reserved output tokens
Compare that total with the model’s documented context window. Do not reserve the entire remaining window for output by default. A practical application usually needs a safety margin for tokenizer variation, message wrappers, tool definitions, and unexpected input growth.
A repeatable token-budget worksheet
- Identify the exact model and API format. Token counts can vary between models, even when the visible prompt is identical. Chat message wrappers, tool schemas, and response formats may also add overhead.
- Count static content. Measure the system prompt, reusable examples, policy text, output instructions, and tool definitions. Store these values separately because they are easy to optimize once measured.
- Count dynamic content. Measure the user request, selected conversation turns, retrieved passages, metadata, and any pasted files.
- Set an output reservation. Estimate the longest acceptable answer rather than the average answer. Structured output needs room for property names, values, punctuation, and escaped text.
- Add a safety margin. Use a documented engineering allowance rather than assuming the estimate is exact. The right margin depends on how variable the inputs are and how costly truncation would be.
- Enforce the budget in code. If the request is too large, remove low-value history, reduce retrieval results, summarize older turns, or reject the request with a useful message.
For implementation, use the tokenizer or token-counting utility recommended for the selected model. A rough character or word estimate is useful during early planning, but it should not be the final control in production. Log the provider-reported input and output usage when available, then compare real usage with your preflight estimate.
Inputs and assumptions
A useful token budget starts with explicit assumptions. Record these inputs in a worksheet, configuration file, or monitoring dashboard:
| Input | What to record | Why it matters |
|---|---|---|
| Model | Provider, model identifier, and API version if relevant | Determines tokenizer behavior, context limit, and pricing rules |
| Context limit | Documented maximum combined context | Sets the hard ceiling for input plus output |
| Static prompt | System instructions, examples, and fixed policies | Repeats on every request and can dominate baseline usage |
| History policy | Number of turns or maximum history tokens | Prevents long conversations from consuming the budget |
| Retrieval allowance | Maximum chunks, characters, or tokens added by RAG | Controls variable input growth |
| Output reservation | Maximum response tokens needed for a valid answer | Protects against incomplete or invalid output |
| Pricing unit | Input, output, cached, or other provider-specific units | Prevents incorrect cost calculations |
Provider comparisons should focus on these dimensions rather than a static ranking. One provider may publish separate input and output limits, another may distinguish cached input, and another may expose different token-counting utilities. Context limits, tokenizer links, and pricing examples can change as models are released or retired, so link directly to the selected provider’s current documentation in your internal worksheet.
For retrieval-augmented generation, count the complete serialized passage sent to the model, not only the source text. Chunk titles, document identifiers, delimiters, permissions metadata, and citation instructions all consume tokens. A reliable RAG implementation should have a maximum retrieval budget and a policy for selecting the most useful passages.
Likewise, tool definitions can be surprisingly significant. Function names, descriptions, parameter properties, enums, and validation rules are included in the request in many APIs. Keep tool descriptions precise, remove unused fields, and measure their serialized form. If your application depends on machine-readable results, pair the budget with schema validation patterns from this structured output guide.
Worked examples
Example 1: A support assistant
Assume a support assistant has the following estimated input:
- System instructions and response rules: 700 tokens
- Conversation history: 1,800 tokens
- Retrieved help-center passages: 2,400 tokens
- Current user message: 180 tokens
- Tool definitions and formatting overhead: 420 tokens
The estimated input is 5,500 tokens. If the application reserves 900 tokens for the answer and keeps a 10 percent planning margin, the working requirement is approximately 7,040 tokens: 5,500 plus 900, multiplied by 1.1. The team should compare that figure with the selected model’s current context limit. If the request approaches the limit, the safest first changes are to reduce old history and retrieve fewer, more relevant passages—not to silently cut the output reservation.
Example 2: A structured publishing workflow
Consider an AI content workflow that extracts article claims into a JSON object. The prompt includes 1,000 tokens of instructions, 3,000 tokens of source material, 600 tokens of metadata, and 500 tokens of schema and examples. The estimated input is 5,100 tokens. The output reservation should account for every claim, source reference, confidence field, and validation message. If the expected maximum output is 1,400 tokens, the unbuffered total is 6,500 tokens.
Instead of assuming the JSON will fit because the source article is short, test the longest realistic article and the largest expected claim set. If validation fails because the response ends mid-object, increase the output reservation or split extraction into smaller batches. A fact-checking workflow can then review claims separately from generation.
Estimating cost
When a provider charges separately for input and output, use:
Estimated cost = (input tokens ÷ billing unit × input rate) + (output tokens ÷ billing unit × output rate)
Insert the current rates from the provider’s pricing page; do not hard-code an old example into a long-lived calculator. For a monthly estimate, multiply the per-request result by expected requests, then add a growth scenario for longer prompts, retries, and peak usage. Keep retries visible because a failed or timed-out request may still consume tokens depending on the API behavior. See the guide to LLM API error handling before treating retry volume as negligible.
When to recalculate
Recalculate the budget whenever an underlying input changes, not only when the application fails. Set a quarterly review as a minimum for production systems, and perform an immediate review when:
- You switch models, providers, API versions, or response formats.
- A provider changes context limits, tokenizer behavior, pricing units, or cached-input treatment.
- The system prompt, examples, tool definitions, or JSON schema grows.
- You add more conversation history, retrieved documents, metadata, or citations.
- Users report truncated responses, invalid structured output, or missing context.
- Usage costs or latency rise without a corresponding increase in traffic.
- You change retry, fallback, summarization, or truncation logic.
Make recalculation operational. Add token counts and estimated cost to development logs, sample production requests for real usage, and alert when input tokens exceed a planned threshold. Test boundary cases: an empty retrieval result, maximum history, a long URL, code-heavy input, multilingual text, and the largest valid structured response.
Finally, maintain a small token-budget record beside each prompt version: model, measured static tokens, maximum dynamic tokens, output reservation, safety margin, context limit, pricing source, and review date. This turns context window management into an ordinary part of AI app development rather than an emergency fix. When prompts need improvement, use a measured prompt A/B testing workflow and compare quality, token usage, latency, and failure rate together.
