RAG only works if retrieval is boringly good

RAG is usually described as a generation pattern. In practice, much of the quality is decided before generation begins.

The model can only reason over the evidence it receives. If the wrong chunks arrive, it may produce a confident answer from irrelevant context, fall back to memory, or refuse a question that the knowledge base could have answered. The retrieval problem is easy to hide because fluent output can disguise it.

search language is not user language

Users do not ask questions in document titles. They use old names, acronyms, error strings, screenshots, and shorthand from a conversation. A retrieval layer has to expand or reinterpret the query without throwing away the exact token that identifies the problem.

The question sent to search is an interface between two vocabularies: the user’s and the corpus’s.

chunk boundaries carry meaning

Chunking is not a preprocessing detail. It decides what evidence can arrive together.

• Documentation benefits from headings and section paths.
• Code benefits from function and class boundaries.
• Tickets benefit from conversation turns.
• PDFs benefit from pages, regions, and layout.
• Policies benefit from clauses and definitions.

A chunk that is semantically similar but missing its scope can be worse than no result at all.

metadata is part of correctness

Vector similarity cannot enforce permissions, tenant boundaries, freshness, or document type. Metadata filters and ranks the candidate set, while citations make the selected evidence inspectable.

user question
  -> keyword anchors
  -> semantic search
  -> metadata filters
  -> rerank
  -> pass evidence to model

Each step can improve relevance, but each step can also remove the only useful source. That makes the pipeline worth measuring as a pipeline rather than treating retrieval as one opaque score.

evaluate the evidence separately

Before judging an answer, ask whether the expected document appears in the top K results. Retrieval and generation fail differently, so they need separate checks. A response can be well written and still be grounded in the wrong policy.

RAG becomes dependable when the unglamorous parts behave predictably: the right source appears, the wrong source is excluded, and the citation leads somewhere a reader can verify. Prompt cleverness cannot compensate for missing evidence.