Running a fleet of Linux servers without losing the thread

Order matters. Look at what listens on the network first, because that is the surface anyone can reach today. Then at what starts at boot, because that is what comes back after a reboot. Then at what consumes, because a full disk causes failures that no application log announces in advance. A server inherited from a previous provider almost always carries a forgotten public key in an authorized_keys file, or a cron job shipping data somewhere nobody watches any more.

None of it counts unless it is written down. An inventory typed into a terminal dies with the tab. The useful minimum is a few lines per machine: role, owner, what runs on it, what you are allowed to restart, and who to call. That document, not an operator's memory, is what holds at three in the morning.

The hardening itself is a short list of directives. Once your keys are in place and proven: PasswordAuthentication no, KbdInteractiveAuthentication no, PermitRootLogin prohibit-password or no, AllowUsers or AllowGroups to name the permitted accounts explicitly, a low MaxAuthTries, a short LoginGraceTime, agent forwarding off by default. On current distributions, put these in a dedicated file under /etc/ssh/sshd_config.d/ rather than editing the packaged config, so the next package update does not quietly revert them.

Moving SSH off port 22 cuts the volume of recorded attempts, not the actual risk: a targeted scan finds the service in seconds. The real benefit is readable logs. What secures a VPS is key-based authentication, banning offending addresses with fail2ban, a firewall that opens only what is needed, and unattended security updates. The OpenSSH documentation and the CIS Benchmarks are the two references worth keeping open while you work.

What determines your exposure is simply this: who opens the connection. A listening port on a public address is scanned continuously, whatever number you give it, and has to be defended around the clock. A connection established outbound from the machine to a rendezvous point cannot be reached from outside; there is nothing there to find. The same reasoning is why an outbound tunnel beats a port forward on a corporate network whose perimeter firewall is not yours to change.

A bastion still has one real advantage: it concentrates access and logs, which makes auditing straightforward. Its weaknesses are equally well known, a single point of failure and an attractive target. An outbound tunnel moves the trust question to the rendezvous point instead: what does it see, and what does it keep? That is the question to put to any remote access tool, a web SSH terminal included. Does it hold your credentials, in what form, and who is able to decrypt them?

Write it during the first run, while you still remember why a flag is there and which error you hit. Fix it during the second, when the gap between what is written and what you actually type becomes visible. A runbook that has not been replayed in six months should be treated as wrong until proven otherwise.

Idempotence is what makes the difference under pressure: a procedure you can rerun without breaking the current state can be read calmly in the middle of the night. In practice that means testing before creating, writing complete configuration files instead of appending lines, and reloading a service rather than restarting it where the software allows. Say what the runbook does not cover, too. The case where the final check fails deserves a line, even if that line only names the person to call.

Three habits cover most of the drift. A naming convention that encodes role and environment, because the most expensive mistake remains the right command on the wrong machine. One inventory that acts as the single source, instead of three files that disagree. And a per-machine history, so that "what changed since last week?" has an answer nobody has to reconstruct from memory.

Operations tooling is then judged on three questions: can the same operation be applied to a group of machines, can you find what was done on a given box, and can you see its current state in the same place? That is the gap Servor works on. The 27 official runbooks it ships (nginx, Caddy, Apache, PostgreSQL, MySQL, Redis, Docker, Podman, WireGuard, Tailscale, SSH hardening, fail2ban, UFW, unattended upgrades, among others) turn routine work into replayable procedures, and every run lands in one unified history with its output, exit code and duration.

The system pieces exist: sudo logging, auditd for sensitive syscalls, journald forwarded to a remote collector so a local compromise cannot erase the evidence. They serve a compliance audit well. They serve a team's daily question less well, which is what was done on this box last night, by whom, and whether it worked. That is an operations need before it is a security need.

Servor records every command in one place regardless of origin, whether typed by hand, run from a runbook, executed in the terminal or proposed by the AI copilot, along with its output, exit code and duration. On top of that sits a hash-chained audit log, write-only at the database level: an entry cannot be modified or deleted. For an agency operating servers on behalf of several clients, that is the difference between stating what was done and showing it.

Access goes through an agent installed on the machine, a single compiled binary published for seven targets (Linux x64 and arm64, glibc and musl, macOS x64 and arm64, Windows x64). It opens no port: the agent establishes the outbound connection, which is what makes it work behind NAT, a corporate firewall or a VPN-only network. It runs as the configured system user and never receives a credential. The web terminal is re-attachable, so closing the tab does not kill the shell, and a command started from the form runs to completion even if the browser drops. The same agent pushes metrics every 15 to 300 seconds and runs checks itself, which makes them independent of any open session: nine check types, from HTTP and TCP to SSL certificates, disk space and custom scripts, with configurable hysteresis so a single blip does not flip a monitor.

That leaves credential handling, the sensitive part of any administration console. The passphrase, the key derived with Argon2id and the X25519 private key never leave the browser; secrets are encrypted with AES-256-GCM, and every command is signed in the browser with an Ed25519 key derived from the vault, then verified on the target machine itself. The direct consequence: the server cannot read your credentials, and a compromised control plane cannot forge an execution. The trade-off is deliberate and worth knowing upfront, because losing both the passphrase and the 24-word recovery phrase makes the data permanently unrecoverable.