Vector stores need product discipline

Vector stores help AI systems remember, but memory is a product feature before it is a database feature.

The demo version is easy. Split documents into chunks, embed them, put the vectors in a database, search by similarity, and stuff the top results into a prompt. That can look good in an afternoon.

The production version is pickier.

chunking decides what can be remembered

Chunking is where a lot of retrieval quality is decided.

A paragraph-sized chunk may work for prose. A table may need row context and column headers. A code file may need function boundaries. A policy document may need section hierarchy. A slide deck may need speaker notes, title text, and image captions. A chat transcript may need turns grouped by topic instead of by token count.

Bad chunks create weird answers. The model retrieves half a policy, loses the exception, and answers with confidence. Or it retrieves a code snippet without the import that explains the type. Or it retrieves a table row without the column name that gives the number meaning.

I like storing chunk metadata that explains the source shape:

{
  "chunk_id": "policy-refunds-v3:section-4:chunk-2",
  "document_id": "policy-refunds-v3",
  "document_version": "2024-07-18",
  "section": "regional exceptions",
  "chunk_kind": "policy_section",
  "text": "Customers in region CA may request manual review..."
}

That metadata is not decoration. It gives retrieval a way to filter, cite, expire, and debug.

metadata is control

Vectors are good at similarity. Metadata is good at control.

A production vector store should usually index more than the embedding and text:

• workspace ID
• document ID and version
• source system
• owner
• permissions
• language
• content type
• created and updated timestamps
• retention policy
• embedding model version
• chunking strategy version

The metadata lets the application ask a narrower question:

const results = await vectorStore.search({
  queryEmbedding,
  topK: 12,
  filter: {
    workspaceId,
    allowedDocumentIds,
    contentType: ["policy", "runbook"],
    archived: false,
  },
})

Without that filter, semantic search becomes a very confident leak.

permissions belong before the prompt

Retrieval permissions are easy to get wrong because the interface feels read-only.

The model is “only” retrieving context. But context can contain customer data, internal notes, incident reports, secrets, salary bands, unreleased product plans, or private support history. If the user could not open the source document, the model should not retrieve its chunk on their behalf.

I would rather duplicate permission metadata into the retrieval index than rely on a fragile join after the fact. The source of truth should still live in the product system, but the vector store needs enough scope information to prevent obviously invalid candidates from entering the prompt.

embedding versions are migrations

Changing embedding models is a schema migration.

The vector dimensions may change. Similarity behavior may change. Multilingual behavior may improve. Code retrieval may get better while policy retrieval gets worse. Distances from one model are not comparable to distances from another model unless the system explicitly accounts for that.

I like storing embedding model version on every record:

{
  "embedding_model": "text-embedding-model-2024-07",
  "embedding_dimensions": 1536,
  "chunker": "policy-section-chunker:v2"
}

Then migration can be staged. Re-embed a sample. Compare retrieval results. Rebuild a shadow index. Run evals. Switch traffic by collection or tenant. Keep rollback possible until the new index has earned trust.

evals should ask about recall, not vibes

End-to-end answer quality matters, but retrieval deserves its own evals.

An answer can be wrong because the model ignored the right chunk. It can also be wrong because retrieval never found the right chunk. Those are different failures.

I would keep eval cases like this:

{
  "query": "Can a Canadian customer get a refund after 30 days?",
  "must_retrieve": ["policy-refunds-v3:section-4"],
  "must_not_retrieve": ["policy-refunds-v2:section-2"],
  "allowed_answer": "manual_review_required"
}

The retrieval score should report recall at K, missed required chunks, stale source usage, permission-filter failures, and hard negative rank. The answer score can sit on top of that.

deletion is part of memory

Memory that cannot forget is a liability.

Documents are deleted. Permissions change. Customers leave. Policies expire. Incident notes become misleading. A source system corrects bad data. The vector store needs a deletion and reindexing path that is as real as the ingestion path.

The ingestion pipeline should answer:

• how do we delete every chunk for a source document?
• how do we rebuild chunks after source edits?
• how do we detect stale embeddings?
• how do we remove data after a retention deadline?
• how do we prove a deleted document no longer appears in retrieval?

That last question matters. “We sent a delete request” is weaker than “the deleted document no longer appears in scoped retrieval tests.”

Vector stores are useful when they are treated like a memory system with rules, not a magical pile of semantic proximity.