The failure is not the alert you missed
Monitoring is usually judged on how fast it notices. That is the wrong metric, and it produces the wrong system: shorter intervals, tighter thresholds, one more check per service, and within a quarter the on-call channel has become weather. People still glance at it. They no longer read it.
The interesting quantity is not detection time, it is how many notifications a human can still take seriously. That budget is small, it is per person, and every alert spends from it — including the ones that turn out to be nothing, especially those. Which gives the thesis of this piece: an alerting system should be slow to shout. Deliberately, measurably slow. The three extra minutes you spend confirming a failure are not lost detection time; they are what keeps the message meaningful when it does arrive.
Everything below applies to any monitoring tool. The last section says what Servor does with it, including what it does not do.
Detection latency is a decision, not a measurement
The arithmetic is trivial and almost never written down: your detection delay is the interval multiplied by the number of consecutive failures required to flip. A check every 60 seconds that flips on the third failure detects in three minutes, not one. Add the notification path and you are at three and a half.
So decide the number first. For each service, ask what delay it can absorb before someone has to know, and derive the pair from that. Doing it the other way round — picking 30 seconds because it feels responsive, then discovering the flapping — is how most noisy setups were born.
Two consequences worth keeping in mind:
- Sampling faster does not detect proportionally faster. It multiplies the number of samples that can fail spuriously. On a normal network, most isolated failures have resolved before the following check — nobody should ever have heard about them.
- The floor is not the interval, it is the propagation. A configuration change, a DNS TTL, a load balancer draining a node: several of the things you monitor cannot honestly report a stable state in under a minute anyway.
Asymmetric hysteresis: slow to shout, slower to say it is over
Hysteresis is two separate numbers: how many consecutive failures declare the outage, and how many consecutive successes declare the recovery. Setting them equal is the default and it is wrong.
Make them asymmetric, with recovery the stricter of the two. A service that answers once and then falls over again is not recovered, it is oscillating — and each oscillation, on a symmetric setting, costs two notifications. Three failures to open and four successes to close absorbs the oscillation entirely, at a cost of a minute or two on the close, which nobody is waiting for at that point.
The same scruple applies one level up. Closing a public incident the moment the check goes green is how you end up reopening it twenty minutes later, and a reopened incident costs far more trust than ten extra minutes of an honest “monitoring” state. Being slow to announce good news is the cheapest credibility you will ever buy.
Flapping that survives a sensible hysteresis is not a tuning problem. It is a real signal that something is genuinely unstable, and the response is a ticket about that instability, not a bigger number in the failure counter.
Thresholds that mean something
Resource alerts are where most of the noise lives, because a resource metric is a proxy for a symptom and people alert on the proxy.
- CPU. An instantaneous reading is meaningless — 100% for twenty seconds during a build is the machine doing its job. Alert on a sustained breach with an explicit minimum duration, and prefer saturation over utilisation: load average compared against the core count says more about queuing than a percentage does.
- Memory. Alerting on free memory guarantees noise, because a healthy Linux box uses everything it can for page cache. What matters is available memory, and above all swap activity and the OOM killer showing up in the journal — those are events, not gradients.
- Disk.A percentage lies at both ends: 90% of a 2 TB volume may be weeks away, 90% of a 20 GB volume is tonight. Reason in days remaining at the observed growth rate, and keep a second, high, non-negotiable percentage as the backstop for the day something writes a log file in a loop.
- Certificates. The most predictable outage in the industry, and still one of the most common. It deserves a warning weeks out — and it deserves not to be a page, because there is nothing to do at 3am about something expiring in 20 days.
And a check that returns 200 while its database is unreachable is worse than no check at all: it manufactures confidence. A health endpoint should fail when its dependencies fail, or it should not be the thing you alert on.
What earns the right to wake someone
One test, applied without sentiment: is there an action, right now, that only a human can take, and does the cost of waiting until morning exceed the cost of the interruption? If the answer is no, it is not a page. It might still be worth recording — that is a different channel.
Three tiers, and everything belongs in exactly one:
- Wake someone. Customer-visible unavailability, data at risk, a resource that will be exhausted before working hours. Short list, and it should stay short.
- Tell the team. A degradation, a self-healed failure, a queue growing at a rate that will matter tomorrow. Goes to a room during working hours, not to a phone at night.
- Record it. Trends, capacity, certificates at 30 days. Lives on a dashboard nobody is obliged to read in the dark.
A useful forcing function: every alert message must contain what to do, not only what broke. If nobody can write that sentence, the alert has no owner and no procedure, and it should be demoted to tier three until it has both. Alerts without an associated action are the largest single source of noise in most setups, and they are the easiest to remove because removing them costs nothing.
Group the consequences under their cause
The second big source of noise is structural rather than statistical. A host goes down and its eight checks go down with it: HTTP, TCP, ping, the SSH check, the disk check, two processes and the certificate. One event, eight messages, sent within the same minute, at which point the recipient stops reading and starts scrolling.
The discipline is to separate two kinds of signal and treat them differently:
- Cause signals — the host is unreachable, the agent has stopped reporting. These are few, and they are the ones that should notify.
- Consequence signals — everything that necessarily fails when the cause is true. These should be recorded and shown, and should stay quiet while the cause is active.
An agent that stops reporting is usually the earliest reliable indication of a host failure, because it stops the moment the machine does, whereas an external check still has to wait out its consecutive failures. Treat it as the cause signal it is.
The same reasoning covers planned work: a maintenance window that does not silence the checks it covers teaches the team that alerts can be ignored during working hours — and that lesson does not stay inside the window.
Repetition is not urgency
Re-notifying every five minutes about an outage everyone already knows about adds nothing except the habit of muting the channel. While a state has not changed, there is no news, and no news should not generate a message. Notify on transitions — into failure, into recovery — and hold everything in between behind a delay.
There is a legitimate case for repeating, and it is a different mechanism: nobody has acknowledged. That is an escalation problem, solved by a human process and a rota, not by a louder monitor. Confusing the two turns a person's phone into a metronome and costs you the acknowledgement you were trying to obtain.
The limit you cannot tune away: one vantage point
A check measures a path, not a state. From a single vantage point, a transit failure between that point and your server is byte-for-byte identical to a failure of your server: the request times out either way. No amount of threshold work resolves that ambiguity, because the information required to resolve it is not in the measurement.
You can still narrow it, by making two independent signals disagree usefully:
- the external check fails and the agent on the machine has gone quiet: the machine is very probably down;
- the external check fails while the agent is still reporting normal metrics: the path is suspect, or the service is failing in a way the host cannot see;
- several unrelated targets fail at once from the same vantage point: suspect the vantage point before you suspect all your servers.
This has a direct consequence for what you write publicly. During the ambiguous window, the honest wording is what you observe — “we are seeing failures reaching the API” — not the cause you have not confirmed. Announcing a cause you have to retract is more expensive than saying less. Our piece on incident communication covers the rest of that vocabulary.
Servor is explicit about this limit: its checks run from a single point. If you need to formally distinguish a transit failure from a service failure, or to measure availability from several continents, that is a service designed for it, alongside rather than instead.
Review the alerts you actually received
Tuning is not a one-off. Once a month, take the alerts of the past period and sort them into three piles: acted on, resolved before anyone acted, ignored. The second pile is your hysteresis backlog — every one of those was a notification the system could have absorbed. The third pile is worse: an alert that is routinely ignored is training, and it trains people to ignore the next one too. Delete it or fix it; leaving it is the one option with no upside.
Two practical notes. Count notifications per person per week, not in total — the total is a management number, the per-person figure is the one that predicts whether the next page gets read. And do the review while the data still exists: fine-grained measurements are expensive to keep, so most tools hold the detail for days and only an hourly aggregate for months. A thirty-second spike is invisible in an hourly average, which is where a good number of confident wrong conclusions come from. In Servor the boundary is 7 days of raw metrics and 90 days of hourly aggregates — take the screenshots during the incident, not three weeks later.
The output of a post-mortem should include at least one settings change: a threshold that fired too late, a hysteresis that was too nervous, a notification that woke someone with no possible action, a public message that aged badly. Without that loop, monitoring degrades quietly while the fleet grows, and the team ends up trusting its instinct over its probes.
What tuning does not fix
If a service genuinely fails several times a week, no hysteresis will make that pleasant — it will only make it quieter, which is worse. Noisy alerting is often an accurate report of an unstable system, and the fix belongs in the system. Equally, no threshold replaces an owner: an alert nobody is responsible for will be ignored regardless of how well it is calibrated. And nothing here helps if the alert cannot get out — a notification path that depends on the infrastructure it watches is not a notification path.
What Servor gives you to work with
Nine check types — HTTP, TCP, ping, SSH, DNS, SSL certificate, disk space, process presence, custom script — with an interval from 30 seconds to 1 hour and hysteresis configured in both directions: failures before flipping, successes before returning. That pair is the main dial described in this article, and it is per monitor rather than global.
Checks can run from the agent installed on the machine, which makes them independent of any browser session being open. The same agent pushes metrics every 15 to 300 seconds (60 by default) and there is a distinct alert for an agent that has become unreachable — the cause signal from the grouping section. Threshold alerts on CPU, memory and disk fire on a sustained breach and do not repeat for as long as the alert stays active. Notifications go to email, Slack, Discord or a signed webhook, with filter rules and an anti-repeat delay; the cause-and-consequence discipline is something you express through those rules, not something the product infers for you.
An incident can be opened by hand or automatically when a monitor flips, and it carries its own update log through to the public status page. Monitors and status pages are included from the Operations plan (100 monitors, 5 pages) and uncapped on the AI plan; the Free plan includes neither.
The wider picture — choosing what to watch in the first place, writing during an outage, running a post-mortem that changes something — is on the monitoring guide. And if the check you are about to add is an SSH check, the SSH hardening piece explains what it is really telling you.