Data cleaning is model work

The prompt is the visible part of an AI system. The data decides how hard the prompt has to work.

Bad input consumes model capacity before the model has a chance to answer the actual question. A retrieval index filled with duplicates, labels that change meaning, and empty fields that hide several different states all turn into downstream behavior that looks like a reasoning problem.

Cleaning is not preparation around the model. It changes what the model is being asked to learn.

Repetition changes the lesson

Duplicates distort frequency.

In retrieval, repeated documents occupy space that could have held a better source. In classification, repeated examples make a boundary look more certain than it is. In an evaluation set, near-identical rows can reward memorization while leaving the harder cases untouched.

Near-duplicates are harder to spot because they often arrive through normal workflows. A support ticket becomes an incident note, then a knowledge article, then a training example. One document appears as OCR, raw HTML, and Markdown. Several alerts describe the same underlying event.

The cleaning decision is not simply “remove duplicates.” It is “which rows represent separate examples, and which rows are different views of one event?”

Labels carry policy

A label is rarely just a string.

If urgent means customer impact in one dataset and executive visibility in another, the model inherits the conflict. If reviewers disagree and nobody records why, the disagreement appears as noise. If a synthetic generator creates both the training labels and the evaluation labels, the score may reflect the generators habits rather than the task.

A classification dataset should make room for:

• definitions for every label
• examples near each boundary
• reviewer disagreement
• underrepresented labels
• taxonomy changes over time

The boundary examples are often more useful than the easy ones. They show where a rule is missing, where a new label might be needed, or where the product should route the decision to a person.

A null is a state

Missing data is usually carrying information, even when the schema does not say what information.

null can mean unknown, unavailable, not applicable, redacted, not collected yet, or accidentally dropped. Treating those states as one blank value forces every downstream consumer to guess. A blank closed_at timestamp might mean an open ticket, a deleted record, a failed extraction, or a field that was never collected.

When the distinction affects behavior, represent it:

owner_state:
  assigned
  unassigned
  owner_deleted
  redacted
  unknown

That gives the model a defined state instead of asking it to infer one from absence.

Watch the source, not only the score

Data sources change without announcing themselves to the model.

A vendor adds a field. A logger truncates messages. A support team changes its summary format. A crawler starts including navigation text in every chunk. A migration changes how nulls are emitted. The prompt remains the same while the input distribution moves.

Basic source checks can make that movement visible:

source: support_tickets
row_count_change: +18 percent week over week
null_priority: 2 percent -> 19 percent
new_status_values: waiting_on_vendor
average_body_length: 1400 chars -> 520 chars
duplicate_rate: 4 percent -> 22 percent

Those are model-quality signals. A sudden increase in duplicates or a sharp drop in body length can explain a quality regression before another prompt experiment does.

Leakage produces flattering evals

Evaluation leakage is often a relationship problem, not an identical-row problem.

The same incident may appear in training and holdout under different wording. A synthetic generator may use the same phrasing in both sets. A retrieval index may include documents published after the evaluation cutoff. A benchmark answer may be present inside a source chunk.

Splitting rows is not enough when the underlying situation is shared. Group by the thing that should remain isolated: incident, customer, document, repository, time window, or source object. Keep generated examples out of the holdout unless generated examples are the thing being measured.

Record the cleaning decisions

Dataset changes need a history that a model-quality review can read.

• removed exact duplicates
• merged near-duplicate incidents
• normalized label names
• split unknown from not_applicable
• removed rows after a cutoff date
• excluded rows with missing source permissions
• fixed broken extraction

Without that record, a metric shift becomes a guessing exercise. With it, a team can connect a change in model behavior to a concrete change in what the model saw.

The model learns the shape of the dataset, including its omissions and contradictions. Cleaning changes that shape, so it belongs in the model workflow.