Proving an eBPF fix on kernels we don't have

There is a special kind of failure where your code never gets to run at all. The Linux kernel reads your eBPF program, considers it, and says no. Nothing crashes. Nothing is harmed. The program simply does not exist on that machine, and whatever it was supposed to measure goes dark.

That is what happened to our zero-touch HTTP metrics layer on a perfectly ordinary Debian 12 box. Kernel 6.1, nothing exotic. The same program loaded fine on 6.8 next door. The check went red, the verifier said "bad CO-RE relocation", and one instruction was marked with the loveliest phrase in the whole toolchain: poisoned by CO-RE.

What CO-RE promises, and what it does not

eBPF programs read kernel memory, and kernel structs change shape between versions: fields move, appear, vanish. CO-RE, "compile once, run everywhere", is the mechanism that makes one compiled program survive this: instead of baking a field's byte offset into the instruction stream, the program carries a note saying "the field called iter_type inside struct iov_iter", and at load time the kernel's own type description, BTF, resolves that note into the real offset for this exact kernel.

When the note cannot be resolved, the loader does not guess. It poisons the instruction, the verifier refuses the program, and the load fails cleanly. This is the right default: wrong-offset reads of kernel memory would produce garbage telemetry, and garbage telemetry is worse than none. But it means portability failures surface as load failures on machines you do not have in front of you.

The failure is safe by design. It is also completely blind: the box simply has no HTTP metrics, forever, until someone asks why.

The struct that moved

The poisoned read walked the iovec iterator inside msghdr, the structure the kernel uses to describe where socket data lives. The error pointed at the field iter_type. The field was innocent.

The real story sat next to it. In kernel 6.4, struct iov_iter was rearranged: a new field, __ubuf_iovec, appeared; the plain iov pointer grew a double-underscore twin, __iov; and, quietly, the values of the iter type enum changed their order. Our program had been written against that newer shape. On 6.1, two of the fields it mentioned did not exist yet, the branch that read them was poisoned, and the whole program went down with it.

So the bug was not a wrong read. It was an assumption that one shape of one struct was the shape, when the kernel had in fact worn three different ones across the versions we support: 5.10 with its old field names and bit-flag types, 5.14 up to 6.3 with iter_type, and 6.4 onward with the rearranged layout and reordered enum.

Teaching one program all three worlds

CO-RE has the tools for this; you just have to use them explicitly. The probe now decides the layout at load time: bpf_core_field_exists answers whether this kernel has iter_type or the older type field with a direction bit mixed in, and whether the iovec pointer is called __iov or iov. bpf_core_enum_value asks the kernel what its enum values actually are, instead of trusting the order they had in the headers we compiled against.

The branches for worlds this kernel does not live in are compiled down to constants the verifier can see are dead, and it prunes them. No runtime cost, no guessing: the program asks the kernel which world it is in and reads accordingly.

**The rule we keep: a kernel struct is not an interface. Every field name, and every enum value, is a question to ask at load time, never an assumption to compile in.**

Proving it on kernels we do not own

Here is the practical problem. The fix had to be proven on 5.10, 6.1, 6.6 and 7.0, and we were not going to install four operating systems at two in the morning. Worse, the obvious shortcut, loading the program in containers, does not work: an unprivileged container cannot create so much as a BPF map, so there is nothing to test against.

The way out is that BTF, the kernel's type description, is just a file, and the loader lets you substitute it. You can load a program into whatever kernel you actually have while computing all CO-RE relocations against the type description of a kernel you do not. Debian publishes its kernel images as ordinary packages; inside each one sits the compressed kernel with a .BTF section, and objcopy --only-section=.BTF carves out exactly the part that answers questions about struct shapes. We harvested the skeletons of 5.10 and 6.1 from the official archive, took 7.0 from a modern machine, and ran the whole matrix on one developer box.

The control experiment is the part worth copying. First, the old program against the old kernels' BTF: it must fail, and it must fail with the exact error observed in the field, the same invalid func unknown#195896080 poison, byte for byte. It did, on both. Against 7.0 it loaded cleanly, which matches the fleet too. Only then does the new program's result mean anything: loaded against 5.10, 6.1, 6.6 and 7.0, all four.

**A fix proven only on the kernel you happen to run is a guess wearing a green checkmark. The failing case has to fail in your harness the same way it failed in the world, or the harness is testing something else.**

The ending

The rollout was an anticlimax, which is the point. Old-kernel machines load the program and their HTTP metrics simply appear; newer machines never noticed anything. The poisoned instruction is still in the toolchain's vocabulary, waiting for kernel 7.x to move something else, and when it does, the load will fail cleanly again, the check will go red again, and the harness with a shelf of borrowed BTF files will already know how to interrogate it.

The takeaway we keep: CO-RE moves breakage from run time to load time, and load time is the right place to break, loudly and safely. Struct layouts drift, enums reorder, and none of that is a bug in anyone's kernel; it is the weather. Ask the kernel what shape it is. And keep the skeletons of the kernels you support within arm's reach, because they weigh a few megabytes each and they turn "works on my machine" into a matrix you can actually run.