Engineering blog

The dependency your scanner cannot see

The scanner said 16 vulnerabilities. Reachability analysis said 30 - and 22 of them were not in any dependency. What manifest scanners miss by construction, and why the binary is the only honest input.

Our agent's source is public, which means it now gets the same automated mail every public repository gets. The first dependency report arrived this week: sixteen vulnerability alerts across four modules. The reflex is to bump four version numbers, close the tab and feel responsible.

We didn't, because we wanted to publish a stronger sentence than "we bumped what the bot told us to bump". The honest sentence turned out to require different work, and the numbers behind it surprised us twice.

The manifest is not the program

A dependency scanner reads your module manifest. That is the entire input: a list of third-party packages and their versions, diffed against an advisory database. It is useful, cheap, and blind to the one dependency every Go program has more of than anything else - the standard library.

That blindness is structural, not a bug. The standard library is not in the manifest, so no manifest scanner will ever mention it. And in Go the standard library is not a thin runtime: it is your HTTP server, your TLS stack, your certificate validation, your template engine, your archive handling. For a program like a monitoring agent, that is most of the attack surface.

So we measured instead of diffing: govulncheck, which builds the call graph and reports only vulnerabilities your code can actually reach.

The result: thirty reachable vulnerabilities. Twenty-two of them lived in the standard library of our pinned toolchain, which had quietly fallen five patch releases behind. HTML templating, the HTTP stack, certificate parsing, TLS, URL handling, tar extraction. Not one of the twenty-two appeared in the dependency report, because not one of them was in a dependency.

The fix for the majority of our findings was not a package bump. It was moving the toolchain pin forward - and treating toolchain patch releases, from that day on, as what they are: security updates for our largest dependency.

Reachability cuts both ways

The same call-graph analysis that found the invisible twenty-two also shrank the visible list. Presence of a vulnerable function in your module tree does not mean your program can ever execute it, and a scanner that only diffs versions cannot tell the difference.

One example we now get to keep forever: a cryptography module we depend on ships an unmaintained OpenPGP package with a known issue and no fixed release. A manifest scanner will flag it until the end of time. The call graph shows our code never imports it - not "we think", but "here is the graph". That is the difference between an alert you silence with a shrug and one you close with a proof.

The directive lies about the toolchain

Then we pointed the same tool at our backend services, and it reported numbers that were dramatically wrong - in a way worth understanding, because nothing warns you.

govulncheck judges standard-library findings against the Go version declared in the module manifest. But the manifest directive is a statement of intent, not a record of fact. Our running binaries had been built with a newer toolchain than their manifests declared; one service's manifest was two major lines behind reality. The scan against the directive reported around thirty reachable standard-library findings per service. The scan against the toolchain that actually built the binaries reported single digits.

The error runs in both directions, and both are dangerous. A directive older than the real build shows you vulnerabilities that were already fixed - alarming, but merely noisy. A directive newer than the real build hides vulnerabilities that are still in the binary - silent, and much worse.

The rule we took away: the version that matters is the one inside the artifact you run. Go stamps it into every binary; go version -m <binary> reads it back. Scan against that, not against a text file that drifts.

One practical footnote that cost us a few minutes: the scanner itself is a Go binary, and one installed with an old toolchain refuses to analyze a tree pinned to a newer one. Rebuild the scanner with the same toolchain you are judging.

Where we landed

Toolchain pinned to the current patch release in the manifests and in CI, so the next drift is a visible diff instead of a silent one. Four modules bumped - the ones with reachable findings, which turned out to be half of what the original report flagged. Every binary rebuilt, every scan re-run against the rebuilt artifacts: zero reachable vulnerabilities across the agent and every backend service, with one permanently flagged package we can prove we never call.

If you run Go in production, the short version is five sentences.

Treat the toolchain as your largest dependency: pin it explicitly, and treat its patch releases as security releases.

A manifest scanner's count is neither an upper bound nor a lower bound on your real exposure - it is a different measurement.

Judge findings by reachability, not presence: the call graph closes alerts with proof and opens ones no bot will ever send you.

Scan the binary you ship, not the tree you read - go version -m tells the truth, the directive tells a wish.

And rebuild your scanner with the toolchain it judges.

Our agent runs with real privileges on machines we do not own, so "zero known reachable vulnerabilities" is not a sentence we get to say on trust. The repository is public and the tool is free - the claim re-verifies the same way the build does: on your machine, not ours.

More from the engine room

Source-available, on purpose

The agent's code is now public under the Elastic License 2.0. Why not MIT, why we refuse to say 'open source', and what the license does and does not mean for the person auditing it.

Read the story

Why your uptime check lies about dead domains

A loopback probe will call a site healthy long after its domain has died. Here is how it fools you, and how to stop trusting it.

Read the story

Signed everything, now build it yourself

Asking someone to run your agent as root on their production box is a big ask. The answer is not 'trust us', it is 'here is how to verify us'.

Read the story

Stories like this one, live on your own fleet.

Three hosts free, forever. Discovery does the tour for you.