Server monitoring: fewer alerts, better ones

The usual failure is a wall of infrastructure probes and nothing that talks to the service itself. A host that answers ping tells you nothing about an exhausted connection pool or a stuck worker. The opposite mistake is just as common: one check on the homepage stops being enough the moment the product depends on several backends. Give every critical path its own probe, and make health endpoints report failure when their dependencies are down. A probe that returns 200 while the database is unreachable is worse than no probe at all.

Slow-burning checks deserve separate handling. They do not measure availability, they buy time: a certificate expiring in ten days, a volume at 88 percent, a daemon that stopped without a sound. Nothing breaks today, everything breaks on a Sunday morning. Run them slowly, once to four times an hour, and route them somewhere other than the on-call rotation.

Thirty seconds is a sensible floor for an HTTP check on a public endpoint; hourly is plenty for a certificate or a disk volume. Sampling faster costs load on the target and, more importantly, costs false positives: the higher the frequency, the more transient network blips you catch that no user ever noticed. On a healthy link, most isolated failures clear before the next run, and nobody should have heard about them.

Resource thresholds follow different rules. CPU pinned at 100 percent for twenty seconds during a build is not an incident; load above capacity for ten minutes is. Alert on sustained breach, with an explicit minimum duration. For disk, the percentage on its own lies: 90 percent of a 2 TB volume buys weeks, 90 percent of 20 GB buys one night. Think in days remaining at the observed growth rate, and keep a second, non-negotiable high threshold as a backstop.

Hysteresis is what kills flapping, the monitor that oscillates between states and produces two notifications a minute. Requiring three consecutive failures to declare an outage and two successes to declare recovery absorbs short dips without meaningfully delaying detection. The asymmetry is deliberate: be slow to shout, and slower still to say everything is fine, because a service that comes back once and drops again has not recovered. The same caution applies before closing a public incident.

The second source of noise is structural. When a host dies, its eight checks die with it and whoever is on call receives eight messages for one event. Separate the cause signal, the silent host or the unreachable agent, from the consequence signals, and suppress the latter while the former is active. Separate the channels too: the pager carries what needs hands on a keyboard now, a team channel carries the rest, and slow trends live on a screen nobody is required to read in the middle of the night.

Two families of probes coexist and they tell different stories. An external probe queries the service the way a user would: it sees perceived availability and knows nothing about the cause. A local probe, run by an agent installed on the machine, sees disks, processes and memory, but disappears together with the machine. Hence the value of an explicit signal when the agent stops reporting: it is often the first trustworthy information about a host failure, and it lands before the surface probes react.

Servor runs its checks from a single point. They can also run directly on the agent installed on the machine, which makes them independent of any open browser session. There are no probes spread across continents. If you need to formally separate a transit failure from a service failure, or to measure availability from several countries, pair Servor with a service built for that. Saying so upfront avoids an unpleasant discovery during the first dispute.

Wording sets expectations. Four states are in common use: we are investigating, we have identified it, we have applied a fix and are watching, it is closed. Each says something different, and jumping to the last one early forces a reopening, which costs far more trust than ten minutes of patience. Write in the language of the customer rather than the language of your infrastructure: outgoing email is delayed, not the queue consumer stopped acknowledging jobs.

Two technical constraints shape the exercise. The page has to live outside the infrastructure it describes, otherwise it goes down at the exact moment it becomes useful, and it has to sit at an address your customers already know, which means your own domain with a valid certificate. Add a push channel as well, double opt-in email or a webhook into your customers own tooling, because nobody thinks to open a status page until twenty minutes are already lost.

That reconstruction assumes the data still exists when you go looking for it. Fine-grained samples are expensive to keep, so most tools retain full resolution for days and only an hourly rollup for months. Know which of the two you are reading, because a thirty-second spike is invisible in an hourly average and a lot of wrong conclusions start there. Capture what matters during the incident, not three weeks later.

The deliverable is not the story, it is the decisions. Every incident should produce at least one changed setting: a threshold that fired too late, hysteresis that fired too eagerly, a notification that woke someone with no action available, a public update that aged badly. Without that loop, monitoring degrades quietly while the fleet grows, and the team ends up trusting instinct over instrumentation.

Checks can run from the agent installed on the machine. That agent is a single compiled binary that opens no inbound port: it initiates an outbound connection, so it works behind NAT, a corporate firewall or a VPN-only network. It pushes metrics every 15 to 300 seconds, 60 by default: overall and per-core CPU, load, detailed memory and swap, disks per mount point, network, uptime, processes. Raw samples are kept for 7 days, hourly rollups for 90. CPU, memory and disk threshold alerts fire on sustained breach and do not repeat while the alert stays active, and a separate alert tells you when an agent has gone silent.

Communication lives in the same place. An incident is created manually or automatically when a monitor flips, moves through four statuses, keeps its update log and notifies subscribers. Public status pages get an immutable unique hostname, a custom URL, a custom domain with an automatic certificate, visual customisation, 90 days of history, and subscribers by double opt-in email or signed webhook. Notifications go out to email, Slack, Discord or a signed webhook, with filtering rules and an anti-repetition throttle.