Anomaly detection for product metrics, without the noise
Reliable anomaly detection for product metrics requires baselines over comparable periods, honest uncertainty estimates, impact weighting and deduplication — a fixed threshold alone produces an alert channel everyone mutes.
Why do generic anomaly detectors fail on product metrics?
Product metrics have no universal normal. A 1% error blip is background noise for one product and a serious incident for another; Monday traffic is not Sunday traffic; a conversion rate computed from three sessions means nothing. Generic detectors and fixed thresholds ignore all of this, so they fire constantly — and teams learn to ignore them.
What works instead is context-aware statistics: compare each signal against its own comparable periods, quantify the uncertainty, weight by how many users are affected, and suppress what has already been reported. Detection is only useful once the change is real, sized, located and worth a human's attention.
The alert channel everyone muted.
Naive detection fails on product data for structural reasons, not tuning reasons — no threshold value fixes them.
A fixed line has no context. The same +1% in error rate means different things at 500 events a day and at 5,000,000 — noise for one product, an incident for the other. A threshold that catches the second will page you weekly for the first.
Product signals are seasonal by weekday and by hour. Compare Monday against Sunday and every week starts with a false "traffic spike"; compare 3 a.m. against noon and every night looks like churn. Detectors that use "yesterday" or a trailing average as the baseline manufacture anomalies out of the calendar.
And ordinary detectors are fragile to their own history: one genuine spike inflates a standard-deviation band for weeks, hiding the next real incident behind the ghost of the last one. Add hundreds of metric × segment combinations tested every day, and pure chance alone guarantees a steady drip of false alarms.
- A conversion of 2 out of 3 sessions is "67%" — and statistically meaningless. Small samples must be gated, not celebrated.
- Hundreds of cells tested daily means false positives are a certainty without multiple-testing control.
- The cost is not the noise itself — it is the mute button. Once trust is gone, real incidents arrive unread.
Compare like with like.
The fix is not machine learning — it is statistics that respect how product data actually behaves.
Baselines over comparable periods. For a daily signal, Muon's findings engine uses the median of the last four same-weekday values — robust to one bad day and immune to day-of-week seasonality. Intraday signals compare same hour against same hour. An EWMA tracks the smoothed level so slow drift is visible too.
Uncertainty, estimated honestly. Rates get two-proportion tests with Wilson score intervals, so that 2-of-3 "67% conversion" never reads as significant. Counts get a Poisson model — is today's number consistent with the expected rate? Continuous values get a robust z-score built on median and MAD (z = (x − median) / (1.4826·MAD), flag at |z| ≥ 3.5), because MAD, unlike standard deviation, does not let a prior spike widen the band and hide the next one. CUSUM catches the slow drifts a point test misses.
Multiple-testing control. With hundreds of metric × segment cells tested per day, raw p-values are a false-alarm generator. Benjamini–Hochberg FDR at q = 0.1 within each metric-day family keeps the expected share of false discoveries bounded — plus a practical backstop: a minimum relative effect (≥10%) before anything surfaces at all.
Naive vs. robust, question by question.
Every failure of a naive detector maps to a specific statistical fix.
| The question | Naive detector | Robust approach |
|---|---|---|
| What is normal? | Yesterday, or a fixed line | Median of the last four same-weekday values; EWMA for the trend |
| Is the change real? | A number crossed the line | Significance tests with uncertainty: Wilson intervals for rates, Poisson for counts, MAD-based z-scores for levels |
| What about last week's spike? | Inflates the average and hides the next incident | Median and MAD ignore the outlier; the band stays honest |
| Hundreds of cells per day? | Hundreds of chances for a false alarm | False-discovery-rate control across the whole family of tests |
| Does it matter? | Every trigger looks the same | Impact = deviation × affected users; tiny cells gated by minimum sample size |
| Same issue again tomorrow? | Alerts again, and again | One open finding per dedup key; novelty decides what leads the digest |
None of this needs trained models. Every number is defensible from a formula — which also means every detection is reproducible and auditable.
Real is not the same as important.
Statistics answer "is this change real?" — a second, deliberate layer answers "is it worth a human's attention?"
Significance alone still overwhelms, because plenty of real changes are trivial. The findings design adds three ideas on top of the tests. Impact: the size of the deviation times the number of affected users, so a −40% swing on a 12-session page ranks below a −18% drop on 9,000 sessions. Novelty: a change that continues an already-open finding updates it quietly instead of re-alerting; the first appearance is what deserves the spotlight. Suppression: minimum denominators (on the order of 100 sessions globally, 30 per segment), muted dimensions and annotated windows keep known noise out entirely.
The result is a short, ranked list — scored by impact, confidence, novelty and severity — instead of a firehose. The goal is not to detect everything; it is to make sure that everything surfaced deserves to be read.
An anomaly without a location is homework.
Detection is half the job — a change becomes useful only once you know where it lives and what moved with it.
"Conversion dropped 18%" is a to-do item. The useful version is located: contribution analysis attributes the movement across browser, platform, page, referrer and release — contribution = (observed − expected) / total delta per segment — so "Safari explains 70% of the drop" is a computed number, not a hunch. Drilling greedily by concentration yields a chain, Safari → iOS → /signup, that ends where the evidence ends.
For rates, the movement is further decomposed into rate shift versus mix shift — separating "Safari users convert worse now" from "we simply got more Safari traffic", two findings with entirely different fixes.
Then the change is placed in time and context: a change-point estimate on the series, the nearest release before it, a cohort test comparing sessions on the old and new version directly, and correlated signals scored on co-timing and co-segmentation — such as a browser-error fingerprint spiking in exactly the affected segment. Only after all of that is a language model allowed to phrase the result, referencing the computed evidence and nothing else.
What statistics can't tell you.
A well-built detector knows the edge of its own competence — and says so.
Correlation is not causation. A release that coincides with a drop, in the same segment, with a matching error spike, is strong evidence — and still not proof. Muon labels these as possible explanations with a confidence score, because establishing causation takes experiments, not co-movement.
And statistics cannot know your roadmap. An intentional deprecation looks exactly like a regression; a pricing experiment looks like a conversion incident; a marketing pause looks like churn. The detector's job is to make sure you know about every change that is real, sized and located — deciding which ones are problems remains yours. That division of labor is a feature, not a limitation.
Questions, answered directly.
Do I need machine learning for anomaly detection on product metrics?
Why not just set thresholds on my key metrics?
How does Muon avoid alert fatigue?
Can anomaly detection tell me why a metric dropped?
Catch what changed. Skip the noise.
Muon's findings engine runs this exact design on your own infrastructure — and the investigation engine locates every change it surfaces.