The alert channel nobody read, and the catalogue that fixed it

The alert channel had four hundred unread notifications and the team had learned, correctly, to ignore it.

That is not negligence. It is the rational response to a channel where a staging deploy, a single 502 from a health check, a CPU spike that resolved itself in ninety seconds, and an actual database running out of connections all arrive looking exactly the same. When everything pages, nothing does. The signal is still in there somewhere, and no human is going to find it at three in the morning.

The worst of them fired most nights. One quiet tenant, a handful of requests an hour, one of them fails, and the error-rate monitor reports one hundred percent errors and wakes somebody up. It was technically true every single time. It was also never worth waking up for.

This is the second article in a series on running a multi-tenant SaaS on AWS at team scale. The first one was about a metric that lies. This one is about a channel that shouts, and the alert catalogue a strong DevOps hire on the team built to replace it. I was an engineer on that team, not the person who did this work, and watching it happen changed how I think about monitoring.

Every monitor has to answer one of two questions

The whole catalogue comes out of a single rule. Before a monitor exists, it has to answer one of these:

1. P1: does a human need to act right now? The service is down, saturated, or losing data. Somebody gets paged, on a phone, at any hour.
2. P2: does a human need to know about this in the next few days? Something is degrading, slow, or trending wrong. It goes to a channel and gets read during working hours.

A monitor that answers neither gets deleted. Not demoted, deleted. This was the hardest part to accept, because every monitor was created by somebody who had a reason. But a P3 tier is a graveyard with extra steps: nobody reads it, and its existence lets you tell yourself you are monitoring something you are not.

The split is not about how serious the underlying issue is. It is about whether acting at 3am changes the outcome. A slow endpoint is a real problem and a terrible reason to wake someone, because nobody is going to profile a query while half asleep.

The P1 catalogue

P1 is saturation and outage, nothing else. These are the conditions where the system is either not serving traffic or is minutes away from not serving it.

Two of those rows deserve a note.

Exit code 137 is the most useful single alert on the list. It is what Linux reports when the OOM killer stops a container, and it is the failure that hides best: the task restarts, the service recovers, the dashboard smooths over it, and the only evidence is a stopped task nobody looked at. Alerting on any occurrence turns a recurring invisible restart into a ticket.

The CPU threshold only means something once the task is sized honestly. This is the direct link to the autoscaling problem from the first article: a single-threaded process on a four vCPU task cannot exceed about 25 percent task-average CPU, so a 90 percent CPU alarm on that service is a monitor that can never fire. Standardising to one vCPU per task did not just fix autoscaling. It made the alert threshold real.

The P2 catalogue

P2 is everything that is genuinely wrong and genuinely not urgent. Same rigour, different destination: a channel, working hours, no phone.

Oldest transaction age is the sleeper on this list. Nothing looks wrong for hours, then autovacuum has not run on the busiest table since yesterday, table bloat is growing, and a query plan flips. Reading it once a week costs nothing and prevents a genuinely confusing incident.

Volume guards, or how to stop a quiet tenant paging you

Back to the monitor that fired most nights. In a multi-tenant system, per-tenant traffic varies by orders of magnitude. A rate on its own carries no information about how much traffic produced it, and one failed request out of one is a rate of one hundred percent.

The fix is a composite condition. The monitor only fires when the rate is bad and the volume is high enough for the rate to mean something:

error_rate > 5%  AND  request_count > 200 over the same 10 minute window

Two hundred is not a magic number. It is the point where a five percent rate is at least ten failed requests, which is enough to be a pattern rather than a coincidence. Pick the floor from your own traffic: whatever makes the numerator big enough that the denominator stops being noise.

The same guard belongs on every ratio in the catalogue: 5xx rate, 4xx rate, cache hit rate, anything expressed as a percentage. A percentage without a volume floor is a random number generator on low traffic. This one change removed most of the night pages, and it removed exactly the ones that had trained everybody to ignore the channel.

Flapping, and the windows that stop it

The other half of the noise was monitors that fired and self-resolved before anyone opened the app. Three fixes, all boring:

Evaluate over a window, not an instant. “CPU above 90 percent for 5 minutes” is a completely different monitor from “CPU touched 90 percent”. The instant version fires on garbage collection, on a deploy, on a batch job. The windowed version fires when there is a problem.

Set the recovery threshold below the trigger threshold. If a monitor triggers at 90 and recovers at 90, anything oscillating around 90 produces an alert storm. Recover at 80 and the oscillation produces one alert. This is hysteresis, and every alerting tool supports it.

Tune by observation, over weeks. Every monitor that fired and self-resolved twice in one night got its window widened, its threshold raised, or its priority dropped to P2. That took about a month of small adjustments. There is no way to get these numbers right on paper, because the right threshold is a property of your traffic, not of the metric.

Only production pages

The last rule is the simplest and it was worth more than it should have been: staging and preprod never page anyone. They alert into a channel, and that is it.

An environment whose entire purpose is to break should not be able to wake a human. Alerts from a place where breakage is the expected outcome are the fastest way to teach a team that alerts are meaningless, and that lesson generalises to production alerts within about a week.

What this did not fix

Two things, both worse than the noise.

One monitor in the catalogue was watching the wrong thing entirely. It tracked a downstream cleanup job as a proxy for whether payments were working, which means it was indirect, lagging by the length of a timeout, and noisy for reasons that had nothing to do with payments. A well-tuned alert on a proxy metric is still an alert on the wrong metric, and that is its own article.

And there is a class of error that no threshold catches: the failure that needs a human to reconcile something by hand, often financial, which is not an outage and not a rate. Those still landed in the log firehose and were found later, by accident. Routing them into an actual work queue is a different problem from alerting.

For the solo, single-VPS version of all of this, where the answer is two containers and exactly one alert rule, see the status dashboard and centralised logs setup and the error tracking that replaced users emailing me about 500s. The scale changes the shape of the answer. It does not change the question.

The takeaway

An alert channel is a shared attention budget, and every monitor spends from it. A monitor that fires without changing what a human does is not free, it is a withdrawal, and enough of them bankrupt the whole channel including the alerts that matter.

Writing the catalogue was mostly deleting. What survived answers one of two questions, carries a volume guard if it is a ratio, evaluates over a window long enough to ignore a hiccup, and comes from production. The result was a pager that goes off a few times a month and gets read every time, which is the only metric an alerting setup actually has.