RAG only works if retrieval is boringly good

RAG is usually sold as a generation pattern, but most of the quality lives in retrieval.

The generator can only reason over the evidence it sees. If the wrong chunks arrive, the model can answer from bad context, answer from memory, or refuse. None of those feels like the product people thought they were building.

the question is not the query

Users ask messy questions. They use old names, acronyms, error strings, screenshots, and language that does not match the docs. The retrieval layer has to turn that into something searchable without erasing the detail that matters.

chunks are product decisions

Chunking is where a lot of RAG systems quietly lose.

Docs want headings and section paths. Code wants function or class boundaries. Tickets want conversation turns. PDFs want pages or regions. Policies want clauses and definitions. One chunking strategy rarely fits all of that.

metadata does real work

Vector similarity is not enough. Metadata filters permissions, ranks fresh over stale, keeps tenants separate, and makes citations inspectable. Without it, the system is just finding similar text and pretending that is enough.

user question
  -> keyword anchors
  -> semantic search
  -> metadata filters
  -> rerank
  -> pass evidence to model

eval retrieval before generation

I would rather test whether the expected document appears in top K than argue about an answer that may have been built on bad evidence. Retrieval and generation need different evals because they fail in different ways.

RAG works when the boring part is boringly good. The right source shows up. The wrong source stays out. The citation points at something the user can inspect. A clever prompt can make a bad retrieval system sound better for a while, but it cannot make the right document appear.