Building Production-Ready RAG Systems: Retrieval, Reranking, Evaluation, and Failure Modes

The familiar prototype—embed documents, retrieve a few chunks, place them in a prompt, and display the response—can demonstrate a flow. It cannot by itself explain why an answer succeeded. A model may answer from parametric memory even when retrieval failed, copy a stale chunk faithfully, or produce a plausible sentence from irrelevant context. The final text hides which evidence entered and which evidence was lost.

Production debugging requires named intermediate records: corpus and index versions, normalized query, applied filters, candidate identifiers and scores, reranked order, selected context, abstention decision, generated claims, and cited evidence. With those records, a missing document is a retrieval defect, a buried document is a ranking defect, and a contradiction despite correct context is a generation defect.

Retrieval-augmented generation does not eliminate hallucinations. It adds an inspectable external-evidence path that can improve grounding only when ingestion, retrieval, context assembly, generation, and validation behave as intended.

On the document side, parsing and normalization preserve useful structure before chunks inherit version, source, status, service, environment, and access metadata. Dense representations and sparse terms are indexing choices, not proof that a document is relevant.

On the query side, normalization must retain exact identifiers such as API-504 while adding any authorized tenant, product, environment, and version constraints. Candidate retrieval seeks recall; merging and reranking improve ordering; context selection applies a token and evidence budget; generation consumes only the selected evidence; and support checks decide whether to answer or abstain.

The evaluation side uses a versioned corpus, query set, relevance labels, metrics, slices, and release policy. Retrieval evaluation asks whether evidence survived and where it ranked. Generation evaluation asks whether the answer used that evidence correctly. These are complementary experiments.

The fictional corpus contains active and deprecated database failover procedures, production and staging API timeout guides, cache invalidation, deployment rollback, authentication-incident, observability, and release-checklist documents. Eleven queries cover exact identifiers, paraphrases, metadata filters, stale competition, distractors, a no-answer request, and an ambiguous metadata constraint.

Every dense_proxy value is hand-authored fixture data. It is a deterministic score proxy used to exercise ranking code; it is not an embedding, cosine similarity measurement, vector-database result, provider output, or production observation. The baseline ranks these proxies directly. The candidate removes deprecated documents, applies fixture metadata filters, combines the proxy with lexical overlap, adds deterministic identifier and service bonuses, and selects the top three above an educational threshold.

The fixture intentionally avoids a perfect candidate. It improves two answerable queries and removes stale selections, but answers an unsupported payroll-key question and loses a relevant authentication guide when an ambiguous environment filter conflicts with document metadata. The critical no-answer regression blocks release.

Define a document contract before choosing a vector store. At minimum, preserve a stable document ID, source URI or owner, version, updated timestamp, status, access scope, service or product, environment, section identity, and a content checksum. Parsing failures and unsupported formats should create visible ingestion records rather than silently disappearing.

Normalization should preserve headings, lists, table relationships, code blocks, warning labels, and links when those structures carry meaning. A table cell detached from its header can become misleading; a rollback command detached from its prerequisite can become unsafe. Store lineage from source document to chunk so corrections and deletions can be propagated.

Corpus quality is operational work. Assign ownership for deprecation, review cadence, duplicate resolution, access changes, and index rebuilding. A newer index over stale source documents is still stale.

Chunk at semantic and structural boundaries: headings, procedures, warnings, table units, code plus explanation, and policy sections. Inherit document metadata and add section-level identity. Overlap can preserve continuity, but excessive overlap creates near-duplicates that consume candidate and context budgets.

Chunks that are too small lose conditions, actor, environment, and version context. Chunks that are too large blend unrelated instructions, weaken lexical specificity, increase reranker cost, and crowd the context window. Tables and code require format-aware parsing; fixed character slicing can split the evidence a query needs.

Token sizes sometimes used in examples are starting hypotheses, not standards. Evaluate alternatives on representative queries and relevant-chunk labels, then inspect failure cases by document type.

Useful filters include document and version identity, source, updated timestamp, service, product, environment, access scope, section, status, and deprecated flag. In the fixture, metadata removes the legacy DB-F01 procedure and separates production from staging API guidance.

Metadata can also fail. A user may supply an ambiguous environment, a document may be mislabeled, or a new version may omit inherited fields. Query q11 demonstrates that a strict filter can remove the only relevant authentication guide. Filter decisions therefore belong in telemetry and evaluation.

Authorization must be enforced by trusted application and storage boundaries using the caller's identity and policy—not merely by asking the model to respect metadata. Filters reduce the eligible corpus only after access has been established.

Dense retrieval compares learned vector representations and can match paraphrases whose words differ. Sparse retrieval uses lexical signals such as term frequency and can preserve exact identifiers, product names, error codes, and uncommon terminology. Neither signal guarantees semantic correctness or operational validity.

Hybrid retrieval combines candidate sets or rankings from complementary retrievers. Elastic documents reciprocal rank fusion as one way to merge full-text and vector results. Anthropic's contextual-retrieval work likewise combines embedding and BM25 signals in a particular evaluated setup. Those are source-specific implementations and results, not a law that hybrid search always wins.

The local simulator performs no embedding. Its dense proxy is supplied by the fixture, while lexical overlap is calculated from visible tokens. This separation prevents an educational score from being mislabeled as a real model output.

Candidate generation is usually broader than final context selection. Dense, sparse, and filtered searches may each contribute documents. Merge by stable chunk ID, retain raw scores and source rank, and avoid comparing unrelated score scales without calibration or rank-based fusion.

A low-recall first stage creates an unrecoverable failure: a reranker cannot promote a document it never receives. Evaluate candidate recall at the configured depth before optimizing final answer style. Inspect exact-identifier, paraphrase, stale-version, metadata, and distractor slices separately.

Increasing candidate depth may improve recall but raises latency, memory, and reranker cost. The right depth depends on measured corpus and query behavior; no fixture threshold here is proposed as an industry standard.

A reranker evaluates a query against a limited candidate set and produces a new ordering. Provider implementations may use cross-encoders or other learned models; the executable fixture uses transparent rules so every score component can be inspected.

The candidate adds lexical overlap, an exact-identifier bonus, and a service-match bonus after metadata filtering. That is educational engineering synthesis, not a claim that the formula is generally optimal. Its value is diagnostic: each feature and threshold is versioned and can be ablated on the same queries.

Reranking adds latency and cost, and candidate size changes both. Define timeout and fallback behavior: preserve the first-stage order, abstain, or route to review depending on consequence. Do not silently return an arbitrary partial order after a reranker failure.

Context assembly chooses evidence under a bounded budget. Deduplicate overlapping chunks, prefer current versions, retain necessary prerequisites, diversify across required subquestions, and preserve provenance. A high-scoring chunk can still be excluded if it duplicates stronger evidence or violates access and status policy.

Retrieving more chunks is not automatically better. Duplicate evidence consumes tokens, stale versions create contradiction, irrelevant context distracts generation, and ordering can change which instruction the model follows. Record both reranked candidates and final selected context so a budget drop is not misdiagnosed as retrieval failure.

A practical policy may reserve space for system instructions and the answer, cap evidence per source, and require a minimum support score. These are application choices to evaluate, not universal constants.

A single end-to-end answer score cannot identify these boundaries. Preserve candidate IDs, ranks, selected IDs, and generated citations. When an evaluator flags an unsupported claim, first check whether supporting evidence existed and was selected before changing the prompt.

The distinction also prevents wasted iteration. Prompt changes cannot recover a filtered-out document, and a larger vector index cannot correct a generator that contradicts supplied evidence.

Define a support gate before generation or before final delivery. If retrieval is empty, scores are below a calibrated boundary, evidence conflicts, access removes required sources, or citations do not support required claims, return an explicit insufficient-evidence state or route to review.

Evaluate no-answer cases separately because answerable-query recall can improve while false answers increase. In the fixture, the baseline abstains on the unsupported payroll encryption-key query. The candidate's lower threshold and lexical match to the authentication guide select irrelevant evidence, so no-answer accuracy falls from 1.0000 to 0.0000.

Those numbers describe one authored no-answer case, not an estimated production rate. A real dataset needs more cases, domain review, prevalence-aware sampling, and separate costs for false answers and unnecessary abstentions.

Recall@K asks what fraction of labeled relevant evidence survived in the top K. Precision@K asks how much of the retrieved set is labeled relevant. Hit Rate asks whether at least one relevant result appeared. Mean Reciprocal Rank rewards placing the first relevant result earlier. nDCG is useful when relevance is graded and order matters.

Metrics depend on relevance labels, query distribution, corpus snapshot, K, and judgment completeness. Precision may look low when unlabeled but useful documents exist; recall cannot be calculated honestly when the relevant set is unknown; MRR ignores additional relevant evidence after the first; and nDCG depends on the chosen gain scheme.

The fixture reports all five for answerable queries and no-answer accuracy separately. Its exact decimals are deterministic results of eleven synthetic records, not benchmark percentages.

The baseline achieves Recall@3 of 0.8000; the candidate reaches 0.9000. Candidate MRR and nDCG also improve, and deprecated-document selections fall from three to zero. Queries q02 and q09 move from misses to hits.

Two regressions remain visible. Query q11 applies an ambiguous shared-environment filter and loses the relevant production authentication guide. Query q10 selects unrelated authentication evidence for an unsupported payroll-key request, dropping no-answer accuracy from 1.0000 to 0.0000.

The fixture policy requires candidate recall not to regress, no-answer accuracy not to regress, and zero critical regressions. These are declared educational policy choices for this fixture—not universal production thresholds. Because q10 is critical, the candidate release is BLOCKED despite stronger aggregate answerable-query metrics.

A retrieval test collection needs a corpus snapshot, query set, relevance judgments, and documented label provenance. Include straightforward cases, paraphrases, exact identifiers, ambiguity, stale-version conflicts, metadata constraints, no-answer requests, and adversarial distractors. Preserve historical failures instead of replacing them with easier examples.

Gold relevance labels should be separated from model-generated judgments. Domain reviewers can adjudicate which documents are necessary, useful, stale, or invalid. If an LLM judge proposes labels, version and calibrate it against human decisions; do not silently treat generated labels as ground truth.

Keep development and holdout queries separate, review leakage from prompts and synthetic generation, and update labels when corpus versions change. Retrieval evaluation is empirical; a tiny fixture demonstrates mechanics but cannot estimate real traffic quality.

Per query, record a request or run ID, corpus and index version, query-representation version, applied access and metadata filters, retrieved chunk IDs, raw source ranks and scores, reranked scores, selected context IDs, stage latency, and answer, abstention, or review outcome. Preserve code and policy versions that produced the decision.

Aggregate retrieval metrics by meaningful slices, plus empty-retrieval rate, stale-selection rate, filter-elimination rate, no-answer accuracy, and latency by stage. Averages should link back to inspectable query records.

Document IDs and scores are usually safer telemetry than full text, but identifiers can still be sensitive. Apply access controls, redaction, retention, and purpose limitation. Do not log retrieved document contents, prompts, or generated answers by default merely because they help debugging.

Access-control filtering must occur in trusted application and data layers with tenant and user identity. Metadata helps express scope but does not make authorization complete. Test cross-tenant identifiers, missing scope, stale permissions, and cache keys that omit access context.

A retrieved document may contain malicious instructions, poisoned content, sensitive data, or stale policy. OWASP notes that RAG does not fully mitigate prompt injection and includes modified repository documents as an indirect-injection scenario. Treat document text as data, separate it from system authority, constrain tools and side effects, validate outputs, and preserve source provenance.

Schemas and system prompts do not neutralize prompt injection by themselves. Defense requires layered controls around ingestion trust, access, retrieval, tool authorization, output handling, monitoring, and incident response.

The fixture does not prove production retrieval quality, real embedding quality, vector-database performance, production latency, scale, user satisfaction, security completeness, or universal optimal thresholds. It does not call an LLM and cannot measure generated-answer groundedness.

Its useful result is narrower: a candidate can improve answerable retrieval aggregates and still be blocked by a critical no-answer regression. The evidence path makes that tradeoff reproducible.

Replace proxies with versioned retriever outputs only after assembling representative queries and relevance labels. Compare dense, sparse, and hybrid candidates on the same corpus snapshot. Calibrate filters, thresholds, reranker depth, and context policy by slice, then add generation and claim-support evaluation as a separate stage.

Add ingestion failure records, deletion propagation, access-aware cache tests, bounded timeouts, reranker fallback policy, structured telemetry, privacy review, and incident feedback. Release decisions should name the policy and case evidence rather than rely on a dashboard average.

This guide owns retrieval-system engineering. The related evaluation guide owns the broader repeatable release gate; the agent guide owns bounded tool orchestration; the architecture-selection guide helps decide whether agentic behavior is necessary; and the Python guide covers typed validation and service foundations.