A retrieval-augmented generation (RAG) system is only as dependable as the knowledge base behind it. This maintenance checklist shows how to update documents, refresh embeddings, remove duplicates, repair ingestion failures, and verify access rules without treating every change as a full rebuild. Use it after content updates, source-system migrations, permission changes, retrieval complaints, and scheduled audits.
Overview
RAG maintenance is the work of keeping a searchable knowledge base aligned with its source systems and the answers users expect. It includes more than uploading new files. A reliable process tracks document ownership, detects stale or duplicate content, preserves metadata, handles deleted material, re-embeds changed text when necessary, and tests whether retrieval still returns the right evidence.
Begin by separating four layers of the pipeline:
- Source layer: the systems where information originates, such as a help center, shared drive, database, ticketing system, or content management system.
- Ingestion layer: connectors, parsers, cleaners, chunkers, metadata extractors, and indexing jobs.
- Retrieval layer: the embedding model, vector index, keyword search, filters, reranking, and access-control checks.
- Generation layer: the prompt, language model, citations or context display, refusal behavior, and answer evaluation.
This separation makes troubleshooting more precise. If a changed policy never appears, inspect source synchronization and ingestion before changing the prompt. If the correct document is retrieved but the answer is wrong, inspect chunking, context selection, or generation. For production monitoring, pair this checklist with the LLM observability guide.
Keep an inventory for every indexed source. Record its owner, source location, document identifier, version or updated timestamp, content type, visibility scope, parser status, chunk count, embedding model version, and last successful indexing time. This inventory is the foundation for repeatable RAG data pipeline maintenance.
Checklist by scenario
When a document is created or edited
- Confirm that the source record has a stable identifier and an authoritative owner.
- Capture the source version or modification timestamp so later jobs can identify changes.
- Extract the document again rather than appending a second copy of the edited content.
- Compare the new text with the previous indexed version. Minor metadata changes may not require re-embedding, but changed searchable text normally does.
- Preserve useful metadata such as product, audience, language, publication status, effective date, and access scope.
- Run a retrieval test using questions that should find the new or revised passage.
- Check that answers cite or identify the current version when your application exposes sources.
When a document becomes obsolete or is deleted
- Confirm whether the source is deleted, archived, superseded, or temporarily unavailable. These states should not be treated as identical.
- Remove or deactivate the old document in the index according to your application’s deletion design.
- Search for duplicate copies, exported PDFs, cached chunks, and alternate URLs that may still retrieve the obsolete information.
- Test a question that previously depended on the document. The application should either retrieve the replacement or clearly indicate that current information is unavailable.
- Record the deletion or deprecation event in the audit log.
When ingestion fails
- Capture the source identifier, job identifier, file type, parser error, and retry history.
- Check whether the failure affects one document or an entire connector.
- Inspect text extraction for empty output, broken encoding, missing tables, repeated headers, or page-order errors.
- Quarantine failed records instead of silently dropping them.
- Retry after correcting the cause, then verify chunk count and retrieval behavior.
- Alert the owner when a failed document is important enough to create a knowledge gap.
When embeddings or retrieval components change
- Record the old and new embedding model or configuration as separate versions.
- Do not assume vectors created by different embedding models are interchangeable. Plan a controlled re-embedding job when the model or vector dimensions change.
- Keep the existing index available until the replacement has passed retrieval tests.
- Compare representative queries across both indexes for relevance, coverage, duplicate results, latency, and filtering behavior.
- Switch traffic only after defining a rollback path and confirming that documents, metadata, and permissions were copied correctly.
For model and prompt changes that affect answer behavior, use a reusable evaluation set rather than relying on a few informal queries. The LLM evaluation dataset guide can help organize those cases.
When permissions or access rules change
- Identify which users, teams, tenants, or roles gained or lost access.
- Update document-level and chunk-level metadata if the application uses either.
- Test both positive and negative cases: an authorized user should retrieve the content, while an unauthorized user should not retrieve or infer it through citations.
- Invalidate stale caches and session-level context where applicable.
- Review filters at query time, not only at ingestion time. An indexed document can remain a security problem if retrieval does not enforce current permissions.
What to double-check
Use this compact audit before marking a maintenance task complete:
| Area | Check | Evidence to retain |
|---|---|---|
| Freshness | Changed and deleted source records are reflected in the index. | Source timestamp, index timestamp, job result |
| Duplicates | Near-identical files, URLs, and versions do not compete in retrieval. | Duplicate report and canonical identifier |
| Parsing | Headings, tables, lists, code, and page order survived extraction. | Sample extracted text and parser logs |
| Chunking | Chunks preserve enough context without combining unrelated sections. | Chunk samples, size settings, overlap settings |
| Embeddings | All searchable text uses a compatible, recorded embedding version. | Model identifier and migration status |
| Permissions | Retrieval filters match current source-system access rules. | Authorized and unauthorized test results |
| Retrieval | Expected documents appear for representative questions. | Query set, ranked results, relevance notes |
| Answers | The model uses retrieved evidence and avoids unsupported conclusions. | Evaluation outputs and reviewer decisions |
Also inspect metadata consistency. A document with the correct text but the wrong tenant, product, language, or publication status may be invisible to valid queries or visible to invalid ones. If your system uses structured output, require maintenance jobs to return fields such as status, source_id, changed_count, failed_count, and needs_review. This makes automated workflows easier to monitor and retry.
Common mistakes
- Appending every update: This creates multiple versions of the same answer. Use stable source identifiers and explicit replacement or deletion behavior.
- Re-embedding without testing: A technically successful job can still reduce retrieval quality. Compare old and new results on known queries.
- Ignoring failed parsing: A green pipeline status is not meaningful if a parser produced empty or distorted text. Validate extracted content samples.
- Trusting timestamps alone: A file can be unchanged while its access rules, linked references, or external dependencies have changed.
- Applying permissions only in the prompt: Instructions to the model are not a substitute for retrieval-time authorization filters.
- Testing only successful questions: Include obsolete, ambiguous, unauthorized, and out-of-scope questions to check refusal and fallback behavior.
- Changing several layers at once: Separate ingestion, embedding, retrieval, prompt, and model changes when possible. Smaller changes make regressions easier to locate.
When a user reports a wrong answer, save the exact question, user scope, retrieved chunks, filters, prompt version, model version, and final answer. Do not investigate only the prose response; the error may begin with missing or unauthorized context. For prompt-related changes, maintain a release record using a workflow such as prompt versioning and release management.
When to revisit
Run a lightweight check after every material source update, failed ingestion job, permission change, user-reported retrieval error, or re-indexing operation. Schedule a broader review before seasonal planning cycles and whenever workflows or tools change. A quarterly review is a useful default for documenting ownership, but high-change or high-risk knowledge bases may need more frequent checks.
At each scheduled review:
- Export the source inventory and find records that have not synchronized within the expected interval.
- Review failed, quarantined, duplicate, and orphaned documents.
- Sample retrieval results across major topics, tenants, languages, and access roles.
- Run the evaluation set and compare results with the previous approved baseline.
- Review embedding, parser, vector database, model, and prompt versions for undocumented changes.
- Confirm rollback procedures and ownership for the next maintenance cycle.
Keep this article with your RAG runbook and update its tool-specific examples when your ingestion framework, vector database, embedding API, document formats, or access-control design changes. The durable principle is simple: every indexed item should have a known source, version, owner, visibility scope, processing status, and verification path. That discipline turns RAG maintenance from an emergency response into a repeatable part of AI app development.