Vector stores need product discipline

Vector stores help AI systems remember, but memory is a product feature before it is a database feature.

The demo version is easy. Split documents into chunks, embed them, put the vectors in a database, search by similarity, and stuff the top results into a prompt. That can look good in an afternoon.

The production version is pickier. Which chunks are allowed to be retrieved? Which version of a document did they come from? Who can see them? Are they stale? Did the answer use the right source? Did retrieval improve the result or merely make it more confident?

A vector database can store nearest neighbors. It cannot decide what memory should mean for the product.

chunks are product decisions

Chunking is where a lot of retrieval quality is decided.

A paragraph-sized chunk may work for prose. A table may need row context and column headers. A code file may need function boundaries. A policy document may need section hierarchy. A slide deck may need speaker notes, title text, and image captions. A chat transcript may need turns grouped by topic instead of by token count.

Bad chunks create weird answers. The model retrieves half a policy, loses the exception, and answers with confidence. Or it retrieves a code snippet without the import that explains the type. Or it retrieves a table row without the column name that gives the number meaning.

I like storing chunk metadata that explains the source shape:

{
  "chunk_id": "policy-refunds-v3:section-4:chunk-2",
  "document_id": "policy-refunds-v3",
  "document_version": "2024-07-18",
  "section": "regional exceptions",
  "chunk_kind": "policy_section",
  "text": "Customers in region CA may request manual review..."
}

That metadata is not decoration. It gives retrieval a way to filter, cite, expire, and debug.

metadata does the boring work

Vectors are good at similarity. Metadata is good at control.

A production vector store should usually index more than the embedding and text:

• workspace ID
• document ID and version
• source system
• owner
• permissions
• language
• content type
• created and updated timestamps
• retention policy
• embedding model version
• chunking strategy version

The metadata lets the application ask a narrower question:

const results = await vectorStore.search({
  queryEmbedding,
  topK: 12,
  filter: {
    workspaceId,
    allowedDocumentIds,
    contentType: ["policy", "runbook"],
    archived: false,
  },
})

Without that filter, semantic search becomes a very confident leak.

permissions have to travel with the chunk

Retrieval permissions are easy to get wrong because the interface feels read-only.

The model is “only” retrieving context. But context can contain customer data, internal notes, incident reports, secrets, salary bands, unreleased product plans, or private support history. If the user could not open the source document, the model should not retrieve its chunk on their behalf.

I would rather duplicate permission metadata into the retrieval index than rely on a fragile join after the fact. The source of truth should still live in the product system, but the vector store needs enough scope information to prevent obviously invalid candidates from entering the prompt.

The final authorization check should happen before prompt assembly. Retrieval should return candidates. The application should decide which candidates can be shown to this actor in this context.

embedding versions are schema versions

Changing embedding models is a schema migration.

The vector dimensions may change. Similarity behavior may change. Multilingual behavior may improve. Code retrieval may get better while policy retrieval gets worse. Distances from one model are not comparable to distances from another model unless the system explicitly accounts for that.

I like storing embedding model version on every record:

{
  "embedding_model": "text-embedding-model-2024-07",
  "embedding_dimensions": 1536,
  "chunker": "policy-section-chunker:v2"
}

Then migration can be staged. Re-embed a sample. Compare retrieval results. Rebuild a shadow index. Run evals. Switch traffic by collection or tenant. Keep rollback possible until the new index has earned trust.

Treating embeddings as invisible derived data is how teams end up with stale memory that nobody wants to rebuild.

evals need retrieval-level cases

End-to-end answer quality matters, but retrieval deserves its own evals.

An answer can be wrong because the model ignored the right chunk. It can also be wrong because retrieval never found the right chunk. Those are different failures.

I would keep eval cases that look like this:

{
  "query": "Can a Canadian customer get a refund after 30 days?",
  "must_retrieve": ["policy-refunds-v3:section-4"],
  "must_not_retrieve": ["policy-refunds-v2:section-2"],
  "allowed_answer": "manual_review_required"
}

The retrieval score should report recall at K, missed required chunks, stale source usage, permission-filter failures, and hard negative rank. The answer score can sit on top of that.

If retrieval misses the required chunk, changing the prompt is the wrong first fix.

memory needs deletion

Memory that cannot forget is a liability.

Documents are deleted. Permissions change. Customers leave. Policies expire. Incident notes become misleading. A source system corrects bad data. The vector store needs a deletion and reindexing path that is as real as the ingestion path.

The ingestion pipeline should answer:

• how do we delete every chunk for a source document?
• how do we rebuild chunks after source edits?
• how do we detect stale embeddings?
• how do we remove data after a retention deadline?
• how do we prove a deleted document no longer appears in retrieval?

That last question is important. “We sent a delete request” is weaker than “the deleted document no longer appears in scoped retrieval tests.”

memory should be inspectable

When an AI answer uses retrieval, the product should be able to show what it retrieved.

That does not always mean showing raw chunks to the end user. Sometimes citations are enough. Sometimes a support engineer needs the retrieval trace. Sometimes the model should expose the source title and section. The right surface depends on the product.

For debugging, I want the trace:

query: "refund after 30 days in Canada"
filters: workspace=acme, content_type=policy
top result: policy-refunds-v3:section-4
score: 0.82
used_in_answer: true

Without that trace, retrieval bugs turn into prompt superstition. People tweak wording because they cannot see that the wrong chunk was retrieved.

vector stores are infrastructure with product rules

The database matters. Index choice, latency, recall, cost, scaling behavior, and operational reliability all matter.

But the product discipline around the database matters more than the demo suggests. Chunking decides what can be remembered. Metadata decides what can be filtered. Permissions decide what can be shown. Embedding versions decide how memory migrates. Evals decide whether retrieval actually helps.

Vector stores are useful when they are treated like a memory system with rules, not a magical pile of semantic proximity.