Notes from
the engine room.
Stories from building a monitoring agent that runs on other people's servers: the bugs, the physics, and the design rules they forced on us. Written by the engineers, one story a week.
Proving an eBPF fix on kernels we don't have
A struct changed shape in kernel 6.4 and our zero-touch HTTP metrics refused to load on older kernels. The fix was ordinary. Proving it across four kernel generations without owning a single one of them was the interesting part.
Read the storyTwo machines, one identity
Clones, templates and restored backups are routine. Monitoring usually answers them by accident: a double host, a gap in the graphs, or a quiet impersonation. We answered on purpose.
Read the storyThe retry loop that sent a thousand alerts
A sixty-second network blip, a write-ahead log doing its job too literally, and the messenger whose throttling hid the storm from us for two hours.
Read the storyThe 56 kbps that wasn't there
An offline router kept a live-looking link on our network map for three days, complete with a traffic number. The number was real - it just belonged to something else entirely.
Read the storyTwenty-seven ways to write an expiry date
RDAP was supposed to make domain data machine-readable. Half the registries never showed up. How we read expiry dates from raw WHOIS without lying to anyone - and what a domain 26 hours from death taught us.
Read the storyThe dependency your scanner cannot see
The scanner said 16 vulnerabilities. Reachability analysis said 30 - and 22 of them were not in any dependency. What manifest scanners miss by construction, and why the binary is the only honest input.
Read the storySource-available, on purpose
The agent's code is now public under the Elastic License 2.0. Why not MIT, why we refuse to say 'open source', and what the license does and does not mean for the person auditing it.
Read the storyWhy your uptime check lies about dead domains
A loopback probe will call a site healthy long after its domain has died. Here is how it fools you, and how to stop trusting it.
Read the storySigned everything, now build it yourself
Asking someone to run your agent as root on their production box is a big ask. The answer is not 'trust us', it is 'here is how to verify us'.
Read the storyThe segfault that turned out to be a dying CPU
Three nights chasing memory corruption in a PHP extension, and the bug was in the silicon.
Read the story