How a monitor should behave on someone else's server

A monitoring agent lives on a machine it does not own, watching work it did not create, and it must never become the thing worth monitoring. That sounds obvious until the day an operator points at their CPU graph and asks why it climbed the moment your agent went on, and you realize the answer is: because of your agent.

When checking a site costs as much as serving it

The setup was a box with a few hundred sites on it, and a check that confirms each site is alive by making a local request to it. Cheap, per site. The trap is the multiplication.

For a modern PHP app, a single request is not free. It boots the framework, loads plugins, connects to the database, and does real work before it can answer even a trivial request. So checking one site costs roughly what serving one real visitor costs. Now run that check against a few hundred sites, frequently, and you are generating the load of a small, steady, self-inflicted traffic spike. On a busy box that adds up to a visible chunk of a CPU core doing nothing but answering your own health checks. The operator sees it. It is right there in their graph, and it has your fingerprints on it.

It got more embarrassing. The host's own intrusion-detection was watching its access logs, saw a stream of identical requests hammering every site from the same source, and started flagging it as suspicious. Our health checks looked, to the machine's own defenses, like an attack. We were both a load problem and, to the box's security tooling, a noise problem.

Behave like a guest, not like traffic

The fix is a set of manners, not a clever algorithm.

Budget the work. Instead of checking every site every cycle, the agent checks a bounded number per cycle and rotates through the full list over time. Each site still gets checked regularly, just not all of them at once, so a box with hundreds of sites sees a steady trickle rather than a thundering herd. On a small box with a handful of sites nothing changes; on a huge one, the load flattens into something the machine does not feel.

Pace the starts. Even within a cycle, do not fire everything at the same instant. Spread the checks out so you are not creating a synchronized wave of framework boots that all land on the same second. A monitor that pulses hard once a minute is worse than one that hums quietly the whole minute.

Identify yourself. Every request the agent makes carries a clear, honest user-agent that says what it is. That does two things. It lets the host's own tooling recognize the checks as monitoring and exclude them from its analytics and its abuse detection, so we stop looking like an attacker. And it is just the polite thing to do: when you knock on someone's door hundreds of times an hour, you should at least say who you are.

The principle worth keeping

The best compliment a monitoring agent can earn is that nobody can find it in the graphs. Its resource use should sit far enough below the noise floor that an operator forgets it is running. The second they can see your footprint, you have changed the thing you were supposed to be neutrally observing, and now every measurement includes your own weight.

So treat the host as someone else's home. Take small, regular, quiet steps instead of loud ones. Announce yourself so the house does not mistake you for a burglar. And hold yourself to a budget tight enough that the owner has to go looking to confirm you are even there. A monitor that shows up in the very metrics it reports has stopped observing the system and started being part of the problem.