Clustering Bot Traffic by JA4 Fingerprint, Not IP or User-Agent

2026-08-31

Problem

We needed to identify and profile clusters of automated/scraper traffic from a large volume of raw HTTP request logs — well past what grouping by IP address or user-agent alone could make sense of.

Symptoms

Elevated traffic to certain paths didn't map to a small number of IPs or user agents — it was smeared across many source IPs and slightly varied UA strings
A single day's worth of HTTP request logs meant tens of thousands of gzip-compressed files to work through
A single request-scoped process had nowhere near enough time or memory to read and aggregate all of it in one shot

Observed

Grouping requests by JA4 TLS fingerprint, instead of IP or user-agent, collapsed traffic that looked like many unrelated sources into a small number of consistent clusters
Each JA4 cluster carried a consistent bot-management score, WAF score, ASN, and request-path enumeration pattern across requests, even though IP and user-agent varied per request within the same cluster
A single Worker invocation attempting to list, decompress, and parse a full day's log files ran into its CPU/wall-clock budget well before finishing, with no state persisted to resume from

Root cause (working theory)

We believe two separate constraints were being conflated as one problem. First, IP address and user-agent are cheap for a traffic source to vary — residential proxy pools and header spoofing make both unreliable grouping keys — while a JA4 fingerprint reflects the client's actual TLS stack and holds far more consistently per source. Second, and unrelated to the grouping logic entirely, a Worker invocation is capped in CPU time and carries no memory across requests, so a full-day scan can't run as a single shot no matter how the grouping itself is implemented.

Why the obvious fix failed

The first version was a single request that listed every log file for the target day, decompressed and parsed each one, and aggregated results in memory before returning. It didn't scale past a small slice of a day's traffic: the CPU/wall-clock budget ran out well before the file list did, and because the aggregation only existed in that request's memory, a timeout meant starting over from nothing rather than resuming.

Fix

Split the scan into bounded batches (150 log files per call) and moved every piece of state that needs to survive between calls — the full sorted file list, how far the scan had progressed, and the running per-cluster aggregation — into a KV namespace kept separate from the log data itself. R2 stayed a read-only, append-only source for the raw logs; KV became the only thing that persisted between invocations. A Cron Trigger re-issues the same scan call on a timer, so a full day's scan completes as a sequence of short invocations that each pick up exactly where the previous one left off.

Verify

Interrupting a scan partway through and triggering it again resumed from the stored cursor rather than restarting, and produced the same final per-cluster aggregation as an uninterrupted run. Aggregated results were also readable at any point mid-scan, since they live in KV rather than only materializing once a run finishes.

Takeaway

When a job is too large for one Worker invocation's budget, don't try to shrink the job to fit — checkpoint it. Keep the durable input (R2, read-only) separate from the durable progress state (a KV cursor plus aggregation), process bounded batches per call, and let a Cron Trigger drive it forward. And when the goal is grouping automated traffic rather than just counting it, group by a signal that's expensive for the source to vary — a JA4 TLS fingerprint holds far more consistently across requests than IP address or user-agent ever will.