Semantic search needs permissions

Semantic search feels magic the first time it works.

You type the thought instead of the exact words, and the system finds the thing you meant. The document used different phrasing. The note had an old term. The support ticket buried the answer in a sentence nobody would have keyworded. Semantic search makes the archive feel less literal.

That feeling is real. It is also where teams get careless.

Production semantic search needs permissions, metadata, deletion behavior, and clear retrieval rules. Otherwise the system becomes a very polite way to leak information.

similarity is not authorization

A vector index will happily find the nearest neighbor.

It does not know whether the user is allowed to see that neighbor unless the system teaches it.

That means access control has to run with retrieval, not after the result is already exposed. If a user searches for “layoff plan” and the system shows a title, summary, or snippet from a restricted document before filtering it out, the product already leaked information.

Permissions need to be part of candidate selection.

For many systems, that means metadata filters:

{
  "tenantId": "acme",
  "visibility": "team",
  "allowedGroups": ["engineering", "security"],
  "sourceType": "incident_doc",
  "sourceId": "doc_1842",
  "deleted": false
}

The search path should narrow the index by tenant, group, source state, and policy before the user sees anything.

metadata is part of relevance

Metadata does more than protect access.

It helps the system rank and explain results.

A chunk from current docs may beat an old forum answer. A document owned by the user’s team may beat a global announcement. A recent incident may matter more than a solved one. A draft may be searchable by its author but not by the whole company. A customer-specific note should never cross tenants.

Useful metadata includes:

• owner
• source system
• created and updated dates
• document version
• tenant
• project
• access groups
• language
• source confidence
• deletion state
• retention policy

Without metadata, semantic search can find the right neighborhood and still behave like a bad product.

deletion has to remove derived state

Deleting a document should remove more than the original file.

It should remove chunks, embeddings, cached summaries, extracted entities, rerank features, and any local or remote index entries derived from that document.

This is easy to miss because embeddings feel abstract. They are still derived from content. They may not reconstruct the source directly, but they can preserve enough signal to create privacy and compliance concerns.

The system should be able to answer:

• which index entries came from this source?
• when were they generated?
• which model generated them?
• where are they stored?
• were they synced elsewhere?
• how do we delete or reindex them?

If the answer is “we do not know,” the semantic search system is not production-ready for private data.

explainability should show the source

A semantic result should not be a mysterious card.

The user needs enough context to understand why it appeared: title, source, owner, updated date, matching passage, and access scope when relevant.

This is partly UX and partly debugging.

If the result is wrong, the team needs to know whether the query was weak, the chunk was bad, the embedding model was poor, the metadata filter failed, or the reranker favored the wrong signal.

I like result records that can be inspected:

type SearchHit = {
  sourceId: string
  chunkId: string
  title: string
  score: number
  sourceUpdatedAt: string
  accessReason: "owner" | "group" | "public"
  matchedTextPreview: string
}

The user does not need every field. The system does.

exact search still matters

Semantic search is bad at some things people care about.

Error codes, issue IDs, stack trace fragments, product SKUs, customer names, function names, and version numbers often need exact matching. A purely semantic system can miss the one literal token that mattered.

That is why production search often becomes hybrid. Use exact search for anchors. Use embeddings for paraphrase. Use metadata for scope. Use reranking to combine evidence.

The product should not make users guess which mode they are in. If a query includes ERR_AUTH_4017, exact matching should have a strong voice.

evals need permission cases

Semantic search evals should include relevance and permission behavior.

Relevance cases:

• paraphrased queries
• old terminology
• ambiguous terms
• exact identifiers
• stale versus current documents
• multilingual queries if supported

Permission cases:

• user can access source
• user cannot access source
• source was deleted
• source moved teams
• source changed visibility
• tenant boundary
• owner-only draft

The scary failure is not “bad result.” It is “good result from a place the user should not know exists.”

reindexing is a product operation

Semantic indexes drift as content changes.

Documents are edited. Permissions change. Model versions change. Chunking rules improve. Metadata gets corrected. The index needs a way to catch up without turning search into a stale memory palace.

I would track index versions and reindex reasons:

• source content changed
• access policy changed
• embedding model changed
• chunking rule changed
• metadata backfill
• deletion request

Reindexing should be observable. How many sources are pending? How many failed? Which search surfaces are using the old index? Can the product fall back while the rebuild runs?

Search quality depends on that maintenance path.

useful search is disciplined search

Semantic search is useful because it lets people find thoughts as well as strings.

But production search needs discipline around the parts the embedding model does not solve: access control, metadata, deletion, hybrid retrieval, explainable results, and evals with permission cases.

The model finds neighbors. The product decides which neighbors are allowed, current, useful, and explainable.

That decision is where production search earns trust.