Embeddings are product plumbing

Embeddings are useful because they make search, memory, deduping, and retrieval feel less brittle.

The demo version always looks magical. You ask a messy question and the system finds the right paragraph even though the words do not match. That is a real win, but it is also the easy part. In a product, embeddings are plumbing. They decide what counts as similar, what gets stored, what gets filtered out, and what happens when the nearest thing is still wrong.

the vector is a hint, not the answer

An embedding is a lossy representation. That loss is the point. The model turns a paragraph, an image, an audio window, or a code block into a vector that preserves some kinds of similarity and discards the rest.

That makes embeddings useful and dangerous at the same time. A refund policy and a cancellation policy may sit near each other because they both talk about account changes. That can help broad search and hurt support automation. A deploy note and an incident review may share vocabulary about logs, services, and owners. That can be great for discovery and awful if the user asked only for incidents.

So I do not treat vector distance as truth. I treat it as a first pass. The retrieval system still needs metadata filters, a second ranking step, citations, or a human question if the nearest match is still vague.

chunking decides what can be found

The most common embedding bug is pretending chunking is just preprocessing.

It is not. Chunking decides the unit that can be found. If the chunk is too small, the result may have the right phrase and the wrong context. If the chunk is too large, the vector turns into a smoothie of unrelated ideas. If the chunk split cuts a code block, table, or policy section in the wrong place, the index can retrieve something technically related and practically useless.

For documentation, I usually want chunks that respect headings. For code, I want the unit to be a function, class, route handler, or test case. For support data, I want to keep the customer message, the agent response, and the resolution separate because each one answers a different query.

The record shape matters because it keeps the source shape visible:

type EmbeddedChunk = {
  id: string
  sourceId: string
  sourceType: "doc" | "ticket" | "code" | "trace"
  title?: string
  sectionPath?: string[]
  text: string
  embeddingModel: string
  embeddingVersion: string
  metadata: {
    owner?: string
    language?: string
    createdAt?: string
    access?: "public" | "team" | "private"
  }
}

That metadata is not decoration. It is what keeps retrieval from turning into a semantic junk drawer. If someone searches current billing docs, a stale migration note may look close and still be wrong. Metadata gives the system a way to say “nearby, but not eligible.”

memory needs a boundary

Embeddings become weird when they are used for memory.

It is tempting to save everything, embed it, and call that memory. The result remembers too much and understands too little. Old preferences, stale plans, abandoned drafts, temporary instructions, and one-off exceptions all become retrievable later unless somebody defines what should expire.

For personal tools, I would rather split memory into a few classes:

preference: durable user choice, user editable
project note: scoped to one project, expires when archived
recent context: short-lived, decays quickly
blocked fact: explicitly excluded from future use

That is not a fancy taxonomy. It is a way to stop the system from dragging last week’s context into every new answer.

This matters more in agent systems because memory becomes authority. If the agent can retrieve a stale note and act on it, the bug is no longer just search quality. It is a permission boundary problem. The fix is lifecycle design, not a bigger vector database.

test the misses, not the demo

Embedding systems are hard to evaluate from the happy path. A few good queries can make the system feel smarter than it is. The real work is collecting misses.

I want a small test set with the query, the expected result, acceptable alternates, and known bad matches:

{
  "query": "how do I rotate the webhook secret?",
  "expected": ["docs/security/webhooks.md#secret-rotation"],
  "acceptable": ["runbooks/incident-response.md#webhook-leak"],
  "bad": ["docs/api/webhooks.md#create-webhook"]
}

That bad field is useful because semantic search often finds the wrong neighbor with confidence. An API reference for creating a webhook may be close to secret rotation. The product still needs the rotation runbook.

I care about Recall@K and nDCG, but I still want to open the misses by hand. If the right document never appears in the top 50, the model or chunking may be wrong. If it appears at rank 12, reranking might help. If it appears at rank 2 but the product chooses rank 1, the scoring logic needs work. If the document is outdated, retrieval did its job and the content system failed.

the map works until it starts pretending to be the territory

I still like the map metaphor. It is a decent way to explain why embeddings help. Nearby points tend to be related. Clusters can reveal themes. Search can finally start from meaning instead of exact wording.

But the map is not the territory, and this one was drawn by a model with its own blind spots. Distance does not equal relevance. Similarity does not equal permission. A retrieved chunk does not equal an answer.

That is why embeddings stay in the plumbing category for me. They are most useful when the rest of the system respects what they are: a fast way to propose neighbors. The product value comes from the parts around them, which is where the real work lives. Chunking, metadata, reranking, citations, review, deletion, and the evaluation set built from the misses are the pieces that make the system trustworthy.

That is a less magical story than the demo. I think that is a better one.