Data cleaning is model work

Data cleaning is model work.

That is easy to forget once the prompt becomes the visible part of the system. The model answer is on screen. The dataset, retrieval index, labels, and eval fixtures sit one layer back, quietly deciding how much work the model has to do to compensate for bad input.

If the input is confused, the model spends capability on confusion.

duplicates teach the wrong frequency

Duplicates are not harmless.

Repeated examples make a pattern look more common than it is. In retrieval, they crowd out better sources. In classification, they make a label feel more stable than it is. In evals, they can make a score look better because the system memorizes the common shape.

Near-duplicates are the annoying version. One support ticket becomes an incident note, then a knowledge article, then a training row. A document appears as OCR, raw HTML, and Markdown. A security alert lands in several slightly different forms. Cleaning has to decide whether those are separate examples or one event with multiple views.

That decision changes what the model thinks reality looks like.

labels are part of the product

Bad labels become model behavior.

If urgent means customer impact in one slice and executive visibility in another, the model learns the conflict. If two reviewers disagree and nobody resolves it, the model learns disagreement as noise. If synthetic labels are produced by the same prompt that shaped the eval set, the system can learn the generator’s habits instead of the task.

For a classification task, I want the label system to answer a few plain questions:

• what each label means
• which examples are near the boundary
• where reviewers disagree
• which labels are underrepresented
• whether the taxonomy changed over time

Boundary examples are the best ones. They show where the model will struggle and where the product needs a rule, a new label, or a human review path.

nulls are not empty by default

Missing data usually has a meaning attached to it.

A null can mean unknown, unavailable, not applicable, redacted, not collected yet, or accidentally dropped. Downstream systems often treat all of those as the same hole. That is how a blank closed_at timestamp turns into “still open” in one place and “missing data” in another.

I want richer states when the difference changes behavior:

owner_state:
  assigned
  unassigned
  owner_deleted
  redacted
  unknown

That makes the ambiguity explicit before the model sees the row.

source drift beats clever prompting

Data sources change under the model.

A vendor adds a field. A logger truncates messages. A support team rewrites summaries. A crawler starts pulling navigation text into chunks. A migration changes null behavior. Nothing in the prompt changed, but the input shape did.

That is why source checks matter:

source: support_tickets
row_count_change: +18 percent week over week
null_priority: 2 percent -> 19 percent
new_status_values: waiting_on_vendor
average_body_length: 1400 chars -> 520 chars
duplicate_rate: 4 percent -> 22 percent

Those are model-quality signals, not just data-quality trivia.

leakage makes evals lie

Cleaning is also how evals stay honest.

Leakage is sometimes obvious. More often it is subtle. The same issue appears in train and holdout. A synthetic generator creates training and eval examples with the same phrasing. Retrieval indexes include documents from after the evaluation date. A benchmark answer gets embedded inside a source chunk.

The question is not only whether rows were split. The real question is whether the underlying situations were split.

For AI systems, that usually means grouping by incident, customer, document, repository, time window, or source object before splitting. It also means keeping generated examples out of the holdout unless the eval is specifically about generated examples.

cleaning deserves a changelog

If cleaning changes model behavior, the cleaning should be recorded.

I want to know what changed between dataset versions:

• removed exact duplicates
• merged near-duplicate incidents
• normalized label names
• split unknown from not_applicable
• removed rows after a cutoff date
• excluded rows with missing source permissions
• fixed broken extraction

That changelog is the difference between understanding a metric shift and guessing at one.

The model learns the mess if nobody cleans the mess first. That is why cleaning is model work. It changes what the model sees, what the evaluator measures, and what the product can honestly claim.