Engineering blog

The 56 kbps that wasn't there

An offline router kept a live-looking link on our network map for three days, complete with a traffic number. The number was real - it just belonged to something else entirely.

A router was unplugged and sitting on a shelf, and our network map showed it connected, with a label: 56 kbps. The device had been unreachable for three days. Its neighbor's ports toward it were down. There was an open critical incident saying exactly that. And still the map drew a confident line between the two routers and put a throughput number on it.

The number was not invented. That is the interesting part. Every byte of it was real, honestly measured traffic — of an interface that had nothing to do with the cable the line claimed to represent.

Where the number came from

The map builds its edges from what devices say about their neighbors: LLDP tables, plus the vendor's own discovery protocol where LLDP is thin. Each entry names a local port and what the neighbor called the far end. To put a number on an edge, the fold looked up traffic counters for the ports on both sides and took the busiest member — a reasonable-sounding rule that survives a counter hiccup on one end.

Here is what that rule did in practice. The dead router's side of the story was collected before it died: sixty-four adjacency rows, frozen in time. Its own port counters read zero — it was, after all, on a shelf. So the fold went looking at the *other* side: the far-end port names the dead router had recorded. And among those names, next to the honest physical ports, sat the neighbor's **VRRP interfaces** — virtual addresses that two routers share so that one can take over when the other fails.

A VRRP interface is not a cable. It is an idea with a MAC address — and the MAC is standardized, 00:00:5e:00:01:XX, the same on every vendor on earth. But it has traffic counters like any other interface, and its counters were very much alive, because the *surviving* router was doing its job. The fold picked the busiest named member, the busiest member was VRRP-10-local at 7,039 bytes per second, and the map printed 56 kbps on a link to a powered-off box.

The bitter footnote: the same console already knew the truth. The neighbor's physical ports toward the dead router had their own port-down incidents, opened fourteen minutes before the router went silent. The map just wasn't listening to them.

Why the corpse looked fresh

The second half of the lie was the timestamp. Why did three-day-old adjacency rows still count as current?

Because of a rule we stand behind: **a failing probe must not erase inventory.** If a device fails one poll — a timeout, a busy CPU, a flaky link — and the monitor responds by deleting everything it knew about that device, then every hiccup wipes your fleet view and every recovery starts from amnesia. So facts survive probe failures, and the agent re-announces what it knows on every discovery pass. Each re-announcement stamped the dead router's neighbor table with a fresh "seen" time.

Two honest rules — "take the busiest member" and "don't erase on failure" — composed into a confident lie. Neither rule was wrong. The composition was.

Classify by data, not by name

The fix had three parts, and none of them is a heuristic on names.

First, link members are now classified by what the devices themselves report. The agent walks one more SNMP column — ifType — and ships a small fact mapping each interface to its type. Ethernet types count as physical. Everything else — VLANs, bridges, tunnels, and VRRP (which additionally gives itself away by its standardized MAC) — is flagged and **excluded from the numbers**. A device that predates this collection simply keeps the old behavior; nothing breaks on an older agent.

Second, a throughput label is printed only when the edge has **exactly one physical adjacency**. If two routers are joined by three cables, no single number describes "the link", and pretending otherwise is how we got here. The line lists its members in the tooltip instead — each port, each protocol that vouched for it, each state — and keeps the label for the case where a label is true.

Third, a dead end of a link now looks dead. If a device's poll has been failing — or all physical members on an edge are down — the line is drawn gray and dashed, with no number, and the tooltip names the open incident instead. And the freshness problem got its balance point: a probe that has been failing for **more than 24 hours** finally retracts the inventory it produced. One bad poll still erases nothing. A corpse no longer wears a fresh timestamp forever.

One vendor quirk earned its own line of code along the way: in LLDP, this router family reports the far-end port as 04_cluster/sfp-sfpplus1 — an interface list, then the port. The name is resolved against the ports we actually collect from the neighbor, so sfp-sfpplus1 matches sfp-sfpplus1 and not a string that exists nowhere.

The ending

The router came back from the shelf a few days later. Its incidents closed themselves, and the map now shows the pair joined by dozens of adjacencies — VRRP members flagged and silent, physical members listed by name and protocol, all up, and **no single number**, because there are several cables and no honest way to compress them into one label.

The takeaway we keep: a map is a set of claims, and every number on it must be traceable to a measurement *of the thing the label points at*. When attribution is ambiguous, show structure, not a number — a tooltip with members is worth more than a confident 56 kbps that belongs to something else. And when two of your own rules are individually right, check what they say when composed. That is where the polished lies come from.

More from the engine room

Proving an eBPF fix on kernels we don't have

A struct changed shape in kernel 6.4 and our zero-touch HTTP metrics refused to load on older kernels. The fix was ordinary. Proving it across four kernel generations without owning a single one of them was the interesting part.

Read the story

Two machines, one identity

Clones, templates and restored backups are routine. Monitoring usually answers them by accident: a double host, a gap in the graphs, or a quiet impersonation. We answered on purpose.

Read the story

The retry loop that sent a thousand alerts

A sixty-second network blip, a write-ahead log doing its job too literally, and the messenger whose throttling hid the storm from us for two hours.

Read the story

Stories like this one, live on your own fleet.

Three hosts free, forever. Discovery does the tour for you.