Long context changed retrieval expectations

Long context changed what retrieval is expected to do.

It did not make retrieval disappear. It made tiny shards feel too cramped.

retrieve units, not scraps

With larger windows, retrieval can select coherent source units instead of paragraph fragments. The hard part moves to eligibility, ordering, and citations: what the model is allowed to see, what it should trust, and where the answer came from.

If the context window is generous, I would rather give the model a whole section, a whole policy, or a whole page than a few sentence fragments that lose the thread the moment they arrive. The retrieval layer becomes more about choosing the right unit than about stuffing the tiniest possible chunk into the prompt.

a simple selection loop

The implementation shape is not mysterious:

def select_context(query, candidates):
    scored = rank_by_similarity(query, candidates)
    ordered = sort_by_source_and_time(scored)
    return dedupe_by_section(ordered)[:8]

The point is not the code. The point is that larger windows give you room to be conservative. You can keep a section intact, preserve citations, and avoid the weird middle state where the model has seen part of a source but not enough to trust it.

what still needs judgment

Long context does not solve retrieval quality by itself. It just changes the questions.

Now I care about:

• whether the source unit is coherent
• whether the order preserves the argument
• whether the citation still points at the right place
• whether the answer used a source that was eligible in the first place

That is why long context feels operational rather than magical. It gives the system more breathing room, but the retrieval layer still has to earn its keep.

The practical win is simpler than the hype: fewer chopped-up prompts, fewer missing references, and fewer answers that look right only because the relevant evidence was spread across too many tiny chunks.