Why your uptime check lies about dead domains

A site can be green on your dashboard and dead to the entire internet at the same time. We found three of them on one busy hosting box, all reporting healthy, all unreachable to a real visitor. The probe was not broken. It was answering a slightly different question than the one we thought we were asking.

The comfortable lie of 127.0.0.1

When you monitor a lot of virtual hosts on one server, the cheap and correct way to check each one is to ask the local web server directly. You open a connection to 127.0.0.1, send the site's Host header, and see what comes back. It is fast, it does not leave the box, and it tests the thing you own: is this vhost configured and serving.

The problem is what it cannot see. The local web server will happily serve a vhost forever, regardless of whether the domain still points at this machine, still resolves at all, or still belongs to your customer. DNS never enters the picture, because you dialed an IP. So a domain whose registration lapsed two months ago, whose zone has gone dark, still returns a clean 200 from loopback. Your check is green. Your visitor gets NXDOMAIN.

The three sites we found had let their registrations expire. The local server kept serving them, their Let's Encrypt certs were quietly rotting toward expiry with nobody to renew them, and every dashboard said fine. The loopback probe was telling the truth about the only thing it tested. We were reading more into it than it said.

Adding the question the probe forgot to ask

The fix is to ask the other half of the question: does this name actually resolve, out on the real internet, to somewhere. So after the local check, we do a system-resolver lookup on each primary name. An authoritative NXDOMAIN marks the vhost as not resolving, and that becomes a real signal instead of a silent rot.

We deliberately do not reach for a third-party "is this domain up" API. The agent resolves names the same way the host itself would, with no external dependency and nothing phoning home. A monitor that quietly calls out to someone else's service to do its job is a monitor you cannot run in a locked-down environment, and it is one more thing that can break or leak.

Two details turned out to matter more than the idea itself.

Rooted names, or your own network lies to you

The first version flapped. A domain that was genuinely dead would "resolve" anyway, then fail, then resolve again, all day long. Two incidents for one dead domain, opening and closing on a loop.

The culprit was the resolver search list. When you look up example.org, a stock resolver may append the search suffixes from resolv.conf and try example.org.your-internal-zone too. If your internal zone answers wildcards, a dead public domain "resolves" through your own infrastructure into an address that means nothing. Rotate through a couple of nameservers with slightly different behavior and you get a flap.

The fix is one character. Look up example.org. with a trailing dot. That is a rooted name, fully qualified, and the resolver is forbidden from appending any search suffix. The internet name has to answer for itself. No more borrowing an answer from your own zone.

The second detail: one failed lookup is not proof of anything. Resolvers hiccup. So a name has to come back NXDOMAIN on two consecutive sweeps before it counts, and the very first failure emits nothing at all. A transient blip can neither open an incident nor resolve one. A dead resolver, if you trust it too eagerly, will paint an entire healthy portfolio as dead in one bad minute, and that is a worse failure than the one you were trying to catch.

The domain that came back as someone else

There is a nastier version of a dead domain: the one that got re-registered by a stranger.

A domain expires. For a grace window the registrar still holds it and the customer can renew. Miss that window and it drops, and a drop-catcher grabs it within minutes and parks it on an ad page. Now the name resolves again, to a perfectly valid server that is not yours, serving a "this domain is for sale" page under its own certificate. Every naive check goes green again. The customer's site is gone and the dashboard is relieved.

You can catch this if you look at when the domain was registered, not just whether it resolves. Renewing a domain does not reset its original registration date; re-registering a dropped one does. So a domain that suddenly shows a registration age of a few days is not a domain your customer renewed. It is a domain that fell out of their hands and into someone else's, and you can flag it the day it happens. Registration dates come from RDAP, the structured successor to WHOIS, and we budget those lookups hard: a handful of fresh ones per sweep, cached for half a day, so a box with a few hundred sites warms up over a couple of hours and then costs almost nothing.

Not every part of the world publishes this. Plenty of country-code registries expose no RDAP and no expiry at all. Where the data is simply absent, we say so per domain rather than inventing a reassuring answer. "We cannot see the expiry for this TLD" is honest. A made-up date is how you end up paging someone at 3am over a domain that was fine.

What to take from it

Every check answers exactly one question, and it is usually narrower than the question in your head. "Is the site up" quietly became "does the local server serve this vhost," and the gap between those two is where a lapsed domain hides for two months. When a check looks reassuring, ask what it literally tested. Then go test the part it left out.