DuckDB made local analytics feel normal

DuckDB is useful in the gap between “a folder of files” and “a warehouse project.”

CSV exports, Parquet files, logs, crawler output, benchmark results, and eval runs often start on disk. The old choices were a database, a notebook, a spreadsheet, or a script that nobody could rerun. DuckDB lets that work use SQL before it requires shared infrastructure.

For example:

SELECT
  model,
  count(*) AS runs,
  avg(latency_ms) AS avg_latency_ms,
  quantile_cont(latency_ms, 0.95) AS p95_latency_ms
FROM 'eval-runs/*.parquet'
GROUP BY model
ORDER BY p95_latency_ms DESC;

The query reads the data where it already lives. There is no import step to justify an early question. If the analysis repeats, save the SQL and consider Parquet for a more stable artifact.

SQL also makes local work easier to review than a notebook with cells run out of order:

CREATE OR REPLACE TABLE runs AS
SELECT *
FROM read_parquet('data/evals/2024-06/*.parquet');

COPY (
  SELECT suite, model, count(*) AS cases,
         avg(CASE WHEN passed THEN 1 ELSE 0 END) AS pass_rate
  FROM runs
  WHERE environment = 'prod-like'
  GROUP BY suite, model
) TO 'artifacts/eval-summary.csv' (HEADER, DELIMITER ',');

That file records what was read, filtered, grouped, and written. It can be rerun by someone who did not sit beside the original analysis.

Local does not mean casual. CSV inference can turn dates into strings, strip leading zeros, or change null behavior between exports. Inspect the shape before trusting the result:

SELECT
  typeof(model) AS model_type,
  typeof(latency_ms) AS latency_type,
  min(created_at) AS first_run,
  max(created_at) AS last_run,
  count(*) AS rows
FROM runs;

AI work produces many useful file-shaped datasets: eval cases, agent traces, tool calls, retrieval experiments, and human review queues. A local loop can capture append-only records, query them while the question is forming, save useful SQL, and promote only recurring metrics into shared infrastructure.

DuckDB does not replace a warehouse. Shared access, access control, scheduled jobs, large datasets, and concurrent writes eventually justify one. It delays that decision until the evidence exists.

The folder is the workspace. The query is the record. The result is something another engineer can explain.