“Everything’s normal” isn’t the same as “everything’s fine”: the blind spot in anomaly detection in media

_{Estimated reading time: 8 minutes}

The Blind Spot in Anomaly Detection in Media Data

We have an anomaly detector monitoring the data. The day our data warehouse stopped reflecting reality, it gave the green light. Why the most costly performance failure is also the quietest, and why defining “normal” is harder than it seems.

I’ve always liked articles with a practical slant—the kind where you can take away knowledge that’s applicable to your own situation. Those who know me and see me ranting on LinkedIn from time to time know that I’ve been particularly persistent lately about the importance of data quality. It’s always been that way, but with AI, it’s even more so, if that’s possible.

I try to lead by example, and as the person ultimately responsible for data quality and its use in AI and other projects, it’s my day-to-day obsession. We have all kinds of controls, tests, monitoring agents, and so on. And you’d like to think that everything is running like clockwork. Obviously, there are always issues… This is the story of one of those issues.

1. The Cliff That No One Saw

Suddenly, a performance channel for a lead-generation advertiser went from about 1,900 conversions per month to zero in our data warehouse. Absolutely zero.

The campaigns hadn’t broken down. The advertiser had migrated its tracking to a new CRM (Salesforce) and, in the process, had redefined what constituted a conversion. Our campaign teams were aware of this, were on top of the situation, and continued to optimize using the correct data: the data coming directly from the client’s CRM. In the real world, leads were coming in as usual. The system was working.

What broke was an internal issue—and a much more subtle one. Our data warehouse was still tracking the old conversion metric (the one that the change had stopped triggering). No one on our end had updated which conversion type should be tracked, and the data warehouse that feeds our reports, our dashboards, and our own models ended up, without warning, tracking a phantom. Reality said “healthy”; our data said “zero.”

And to catch exactly that, we have an anomaly detector: to detect silent discrepancies between what’s actually happening and what the data reflects. A metric that drops for no apparent reason is a textbook use case. It didn’t catch it. Every morning, it checked the account, and every morning it issued the same verdict: everything normal.

And, technically, he was right. That’s the problem.

Because in performance marketing, a drop to zero is the most expensive failure there is—and, at the same time, the quietest. A spike in spending or a skyrocketing CPL scream out: they stand out on any dashboard, grab your attention, and trigger a notification at nine in the morning. A zero doesn’t. A zero doesn’t raise its voice. The curve flattens out, the cost per lead can no longer be calculated, and the screen remains… calm. And it’s very easy to mistake that calm for good health.

Let’s be clear, then, about what this story is really about: the campaign management didn’t fail, nor did the client stop selling. The safety net failed—the system that was supposed to alert us that our data no longer reflected reality—and it failed in the worst possible way: silently, by giving the green light. This article is about why building that safety net isn’t, at its core, a problem of statistics, but rather one of defining what “normal” means; and how that poorly defined concept leaves your system blind to precisely what matters most.

2. The real problem isn’t the statistics; it’s defining what “normal” means in media data

When people think of an “anomaly detector,” they usually picture statistics: moving averages, standard deviations, z-scores, and bands that trigger when a number falls outside the expected range. And that part is… the easy part. The math needed to say “this is three standard deviations below normal” fits into just a few lines of code.

The hard part—what really determines whether your system works—is answering a much more uncomfortable question: What is “normal” for this metric? And in media, “normal” is a moving target. Campaigns launch and pause, budgets rise and fall, there’s seasonality, slow weekends, Black Friday, and platforms that report a day late. Half of what a naive detector flags as an “anomaly” is nothing more than just another Tuesday.

And that’s the catch. For a detector to be usable, you have to filter out that noise: if it goes off ten times a day over things that don’t matter, no one will pay attention to the eleventh time—which actually did matter. So you fine-tune it to be accurate, so that it only alerts you when it’s reasonably certain. And in doing so, without realizing it, you build a system that’s deaf precisely to the most serious errors: the ones that don’t look like “a number outside the range.”

It’s also important to distinguish between two tasks that are constantly confused. Monitoring the health of the campaign is something people already do. The detector’s job is different—and more subtle: monitoring the integrity of the data and alerting you when your data warehouse no longer reflects reality. These are different things, and the most common mistake is to assume that the former covers the latter. It does not. In our incident, the campaign’s health was perfect; what had broken was the mirror.

What follows is a list of specific ways in which that mirror shatters. Six blind spots, divided into two categories: the things the system fails to see (false negatives: the actual failure occurs, and no one notices) and the things the system invents (false positives: the alarm goes off when nothing is happening). Let’s start with the first ones, which are the ones that cost us the incident.

3. The Anatomy of a Blind Spot in Campaign Monitoring

Two axes and four cells, with almost all the danger concentrated in a single column:

	The value decreases (e.g., 100 → 20)	The value goes to zero / disappears
Volume Monitor (Absolute Threshold)	✅ Detects: “today < threshold”	❌ Blind: compares with the current threshold, not with its own history; and a missing row is not visible
Efficiency Monitor (Ratio: CPL / CPA)	✅ Detects: “CPL spikes”	❌ Blind: CPL = NULL/∞, no data → no alarm

The right-hand column (the drop to zero) is the blind spot that no naive monitor watches. That’s where our incident occurred.

The strength of the chart is that both indicators fail in the same column for different reasons: the efficiency indicator because its ratio becomes undefined, and the volume indicator because it compares today’s value to a fixed threshold rather than to the historical data for that same channel. Let’s go through it cell by cell.

What the System Doesn’t See

The blind spot of zero. Most of the metrics that really matter in performance are ratios: CPL, CPA, cost per acquisition. And a ratio hides a trap: it divides by the result. As long as there are conversions, the CPL goes up or down, and a tracker follows it without any problem. But when conversions drop to zero, the CPL doesn’t become “bad”—it becomes undefined: a division by zero, a NULL, an infinity. A system monitoring “Has the CPL skyrocketed?” doesn’t see an alarming value; it sees a lack of data. There’s nothing to compare it to. The problem isn’t that the alarm doesn’t go off—it’s that it has no way to go off. That’s exactly what happened to us: conversions in the store were zero, so the CPL disappeared, and with it, any signal.

The trap of “the present, not the past.” “Okay,” you think, “I’ll keep an eye on the volume: if today’s conversions fall below a certain threshold, I’ll send an alert.” Better, but not enough. A fixed threshold looks at today’s value in isolation and can’t distinguish between a channel that has plummeted to zero and one that has always been close to zero because it barely generates leads—and in that case, that’s normal. For today’s zero to mean anything, you have to compare it to that same channel’s past: 1,900 → 0 is an emergency; 3 → 0 is just another Tuesday. The signal isn’t in the number itself; it’s in the difference from its own history.

“Disappearance” is not the same as “low numbers.” There’s a more subtle distinction. Sometimes the problem isn’t that a row is zero, but that the row ceases to exist. If the channel disappears from the feed, there’s no “0” to watch for—there’s nothing there. And almost all detectors reason based on what’s present; they scan through rows and evaluate them; what isn’t there isn’t evaluated. Detecting an absence requires reframing the question: no longer “Is this number an outlier?” but rather “What did I expect to see that isn’t there?” This requires maintaining a list of what should appear and monitoring streaks of zeros or silence.

The deployment gap. The most mundane of all—and therefore the most common. A detector only catches what happens while it’s looking. If you build it—or fix it—after an incident, it won’t find it: it’s already happened. Looking ahead isn’t enough; you also need a retrospective scan of the history, searching for spikes and changes in levels that occurred before the system itself existed. Monitoring without memory is blind to its entire past.

What the system comes up with

The false positive due to freshness. Now, the opposite scenario. Suppose you get everything above right and set up a detector that’s sensitive to drops to zero. Morning comes, a connector is halfway through syncing—which is common: some integrations take hours, or fail and retry—and half a day’s worth of data still hasn’t come in. To your detector, a perfectly healthy channel appears to have crashed. If you raise the alarm, you’re crying “wolf!”; and the third time the wolf doesn’t come, you lose credibility. The only solution is to model the data’s freshness: determine whether a day’s data is complete before judging it, and keep quiet—or mark it as provisional—about what’s still loading.

Not every zero is a mistake. And finally, the most subtle of false positives. Sometimes a zero is… correct. For example, attribution is passed to the ad server, and the platform’s native conversion rate drops to zero by design. A line is intentionally paused. A metric is no longer measured because it no longer applies. A detector that treats every zero as an emergency will end up triggering alerts in response to deliberate decisions, and every false positive erodes that trust that is so hard to build. The system doesn’t just have to detect zeros—it must have a way of knowing which zeros were expected.

The Six Blind Spots, at a Glance

Four of the six are things the system doesn’t see; two are things it makes up. And the timeliness of the data lies at the heart of both—it’s what distinguishes a “broken campaign” from “data that hasn’t come in yet.”

4. What We’re Changing: Changes That Are Being Rolled Out Widely

The good thing about having a blind spot map is that the solutions are no longer improvised: each fix addresses a specific issue. None of them are particularly sophisticated; the hard part was figuring out what needed to be fixed.

Compare against the high for the day (or all-time high), not against a fixed threshold. A direct response to the blind spot of zero and the “current” trap: if you measure the drop using the greater of today’s value and the channel’s baseline as a reference, a 1,900 → 0 triggers the alarm, while a 3 → 0 stays where it should—silent.

Explicit detection of missing values / runs of zeros. Change the question from “Is this number anomalous?” to “What value did I expect to see, and it hasn’t appeared for N days—or is it zero?” Without this, missing values (the quietest kind of error) are never detected.

Modeling data freshness. Before evaluating a day, check to see if it is complete. Data that is still being loaded is marked as provisional and does not trigger an alert. This is what distinguishes a “broken campaign” from “the connector hasn’t finished,” and it prevents false alarms.

Retroactive scanning upon deployment. When starting up or replacing a detector, don’t just look ahead: go back through the history to identify the steps that led up to its existence. Monitoring should be built with memory from the start.

A person at the end of the loop. No flood of alerts: a daily summary, with severity levels, that arrives before the team starts their day. The true KPI of a monitoring system isn’t how many anomalies it detects, but how much trust it inspires (and trust is earned by being accurate and honest about what it doesn’t know).

5. What We Take With Us (and this applies to anyone)

Beyond our specific case, here are a few ideas that can be applied to any system that monitors media data—or data in general:

“Everything’s normal” doesn’t mean “everything’s fine.” A quiet dashboard doesn’t prove that everything is healthy; it may just mean you’ve stopped tracking.

Watch for the absence of what’s good, not just the presence of what’s bad. The most costly error is almost never a bad number—it’s a number that should be there but isn’t.

Every ratio hides a denominator that can trip you up. Keep an eye on the numerator and denominator separately—not just the quotient.

Compare everything to its own past and context, not to a universal standard: what is “normal” in one channel is abnormal in another.

A monitor fails in two ways (not seeing what’s happening and seeing what isn’t happening), and data freshness lies right in the middle of the two.

Treat your monitoring system like a product: it has its own bugs, regressions, and blind spots. The day you assume it’s “done” is the day it starts lying to you.

6. The safety net, and why it matters more and more

It’s tempting to read all this as a technical anecdote: a conversion mapping that didn’t propagate, a detector that didn’t catch on. But the issue is bigger than that, and it’s only getting bigger.

We’re automating more and more layers of media operations: bidding, budget allocation, optimization, and even quality control for the setup. And all of these automated processes draw from the same source: the data in the data warehouse. The moment that data ceases to reflect reality—silently, as happened to us—it’s not just a report that’s wrong: it’s every automated decision that depends on it. The monitoring layer is the safety net for everything else, and its blind spots cease to be a local problem and become a systemic risk.

That’s why the work doesn’t end with detecting zero-hour outages. The next step is to make the network smarter: baselines by entity, thresholds that account for seasonality, alerts that not only indicate what went wrong but also pinpoint the cause, and—on the horizon—detection linked to an automated response.

For now, all it takes is a change in mindset. The next time a dashboard looks normal—with everything in the green and everything calm—it’s worth asking the uncomfortable question: Is everything okay… or are we missing something?

Alex Masip Data & Martech Director MIO One