AI Systems

Threat analytics is time series work too

2 min read

A security event is often only half a sentence. Timing finishes it.

One login failure is ordinary. A burst against dormant accounts at 2:13 AM is different. A PowerShell command may be normal on a build host and strange on an executive laptop.

Local baselines beat one global baseline. Users, service accounts, build agents, laptops, Kubernetes nodes, and cloud workloads have different rhythms:

entity: user:jlondon
feature: failed_login_count
window: 15 minutes
baseline: median by hour-of-week over 30 days
current: 18
expected: 0-2

The window is a product decision. One minute catches bursts. A day catches slow guessing. Distinct countries over a week may show travel or credential sharing. Alerts should expose the window that produced them.

Seasonality is context, not safety. Business rhythms can explain a spike while attackers slowly change the baseline. Track drift with annotations:

feature: outbound_bytes_to_new_domains
entity_group: engineering_laptops
baseline_period: previous_30_days
candidate_period: last_7_days
change: +340%
annotation: new telemetry exporter rollout began 2025-05-01

Sequences matter too. A new-device login followed by a privilege check and mailbox-rule creation within 20 minutes is a different signal from any event alone.

Late data means event time and ingest time need separate fields:

{
  "event_time": "2025-05-07T03:14:22Z",
  "ingest_time": "2025-05-07T03:18:41Z",
  "source": "endpoint",
  "entity": "host:build-17",
  "event": "process_start"
}

Measure detection latency, ingest latency, alert volume, analyst reopen rate, false positives by entity group, missed incidents turned into regression tests, and drift annotations.

The question is not whether an event happened. It is whether it belongs here, at this time, after these events, for this entity.

Jeremy London

About Jeremy London

Engineering leader and builder in Denver. I write about AI platforms, agents, security, reliability, homelab infrastructure, and the parts of engineering work that have to survive production.