Data contracts are where AI quality starts

Model quality work starts before the prompt.

That is easy to forget because prompts are visible and contracts are not. When a model produces a bad answer, the instinct is to rewrite the instruction. Be more precise. Be more careful. Cite sources. Prefer fresh data. Sometimes that is the fix. Often the prompt is carrying a data contract that should have existed upstream.

what the model is allowed to assume

A data contract says what the downstream system can trust.

For an AI feature, that usually means field names, field types, null semantics, freshness expectations, ownership, permission metadata, deletion behavior, label meaning, and allowed enum values. It is not paperwork. It is the list of assumptions the model is allowed to make.

ticket.id: stable unique identifier
ticket.body: latest customer-visible message body
ticket.priority: one of low, normal, high, urgent
ticket.account_tier: nullable only when account is deleted
ticket.updated_at: source system update time
ticket.permissions: users and groups allowed to view source

If that shape is not defined, the model becomes the place where every ambiguity lands.

nulls and labels need a contract first

Nulls are where weak contracts show up fast.

Unknown, unavailable, redacted, not applicable, not collected, and deleted are different states. A prompt can try to explain all of them after the fact, but the data should already carry the difference.

owner_state:
  assigned
  unassigned
  deleted
  redacted
  unknown

Labels need the same treatment. severity, priority, risk, and status are not just strings. They are history, policy, and process wrapped into one field. If a model or evaluator depends on them, the label needs an owner, a definition, and a migration path.

freshness is part of the shape

AI systems usually move through a chain of derived data: source table, export job, parser, chunks, embeddings, index, retrieval result, prompt context.

Any one of those can be technically successful while the result is stale. A useful contract should say how fresh the consumer can expect the data to be:

source: customer_docs
freshness_slo: indexed within 15 minutes of source update
delete_slo: removed from index within 5 minutes of source deletion
permission_slo: permission changes reflected before retrieval

If a user asks about a policy update and the retrieval layer serves yesterday’s chunk, the model may answer fluently and still be wrong.

permissions have to travel

Any data that can be retrieved, summarized, embedded, or used as context needs permission metadata attached to it.

If the source application enforces access but the downstream index drops the boundary, the AI feature can still expose private data. That is the failure mode a contract is supposed to prevent.

source_object_id: required
tenant_id: required
visibility: public | tenant | group | user
allowed_group_ids: required when visibility = group
allowed_user_ids: required when visibility = user
permission_version: required

That does not solve every authorization problem. It does keep the private-data path visible.

checks belong before the model

Contracts are useful when they are executable.

A few checks can catch the boring regressions that make AI systems look flaky:

required field missing
enum value unknown
timestamp older than freshness limit
permission metadata absent
duplicate source object ids
chunk without source hash
label distribution moved beyond threshold

Those checks should fail loudly enough that the pipeline gets fixed instead of the prompt getting longer.

evals are downstream of contracts

Evals depend on stable input assumptions.

If the eval set comes from undocumented fields, stale permissions, or labels that changed meaning, the model is being graded against a moving target. A contract gives the eval something concrete to reference:

eval_set: support_escalation_holdout_v4
source_contract: support_ticket_v7
label_contract: escalation_risk_v3
created_from_snapshot: 2025-03-01
excluded_after: 2025-03-01
permission_model: tenant_group_v2

When the contract changes, the eval may need to change too.

The broader point is simple. The model is only one piece of the system that produces an answer. The contract decides what the inputs mean. If that layer is weak, every later layer spends time compensating for it.