Bot Metrics

Overview

Bot Metrics is an always-running collector that tracks public Twitch channels and serves a password-protected dashboard on the local network. It records stream viewer counts, users in chat, and the residual between them, and it can run a lower-frequency scan that samples the chat roster and estimates how much of an audience looks coordinated.

The detection is statistical, not learned. There is no model and no training. It clusters account-creation dates and username shapes and scores the result, and the restraint in that design is the point.

The problem

“Is this audience real?” is not answerable from public data, and most tools that claim to answer it are guessing confidently.

The available signal is thin. Twitch’s public GraphQL surface cannot observe viewer sessions, and it cannot reliably distinguish a raid, a shared chat, an embed, or legitimate local automation from coordinated inauthentic activity. Follow-wave, message-content, device, IP, and ASN signals all require moderator OAuth, EventSub, or first-party telemetry.

So the honest problem was not “detect bots.” It was: given genuinely weak evidence, what can be said, and how do you keep the system from overstating it?

Detector v4

The detector treats unusual behavior as aggregate cohort evidence rather than as a verdict on anyone.

Two signals are primary. Dense account-creation periods: an implausible cluster of accounts registered in the same narrow window. Large normalized username-template clusters: accounts whose names collapse to the same generated pattern. Each must clear both an absolute threshold and a minimum share of resolved profiles before it counts at all.

Everything else is supporting context only. Recurrence across prior scans, cross-channel overlap, and a robust viewer-to-chat residual are visible in the dashboard but cannot produce a suspicious estimate on their own.

A high score requires both independent primary signals. One primary signal alone is capped at elevated. Small samples are labeled Insufficient sample, and “Not flagged” explicitly does not mean verified human.

Making the score auditable

A 0–100 risk score that a reader cannot interrogate is worse than no score. The dashboard reports the points, the observed values, the thresholds they were measured against, whether each signal acted as primary or supporting, any bonuses, and any cap that was applied.

Confidence is reported separately as two coverage figures. Roster coverage is the unique accounts observed against Twitch’s reported users-in-chat total; profile coverage is the share of discovered non-chatbot accounts whose creation date resolved. A high score at low coverage is visibly a weak result.

Recalibrating the detector meant migrating existing records: v2 and v3 estimates are rescored during migrations 11 and 13, so history is not silently mixed across scoring generations.

Privacy as a constraint on the design

The system is built so that it cannot produce a per-account verdict, rather than merely choosing not to.

Twitch usernames are processed only in memory and never enter SQLite, logs, API responses, or exports. Raw logins and per-account dates are discarded after aggregation; the database keeps only month-level creation totals and cohort estimates. For time-based evidence it stores one non-reversible HMAC presence key per account and channel, keyed by an external session secret, with first and last seen times and a scan count. Presence rows expire after seven days.

Individual-level detection was not an unimplemented feature. It was ruled out.

What I directed versus what the model produced

I designed the detector’s signal hierarchy, the two-primary-signal requirement, the coverage reporting, and the privacy model that constrains what the schema can store. AI coding tools wrote much of the collector, dashboard, and migration code against that design. The decision to refuse identity-level classification, and to weight the residual conservatively rather than fabricate signals the data cannot support, is mine.

Result

The service runs continuously against any number of channels, each with its own polling interval, staggered starts, bounded exponential retry, and a persistent, reorderable scan queue. It is deployed on the LAN behind password authentication and is deliberately not exposed publicly.

What I learned

Building a detector taught me more about evidence than about detection. Deciding which signals are strong enough to accuse, what a score obligates you to disclose, and what the system should be structurally unable to say turned out to be the substance of the work.