The segfault that turned out to be a dying CPU

For three nights we were sure our own code was corrupting memory.

php-fpm was segfaulting under load on one of our development boxes. Eighty-seven crashes in forty-six hours on the busy one. A second box crashed in under a minute the moment we pointed ab at it. The signature never changed, which felt like a lucky break: a crash deep in span serialization at request shutdown, reading a class constant, on a value whose type tag was garbage and whose "pointer" happened to equal the small integer of the constant it was about to read.

That last detail is the kind of thing you learn to love. A value that holds a tiny integer where a pointer should be is not random noise. It is a poisoned struct: something wrote over it earlier in the request, and the crash only surfaces at the end, in the most constant-heavy code, because that is the first place unlucky enough to dereference the corpse. The place a program dies is rarely the place it was killed.

So we did the obvious thing. We blamed ourselves.

The differential that proved nothing

We ship an optional deep-tracing layer for PHP. The agent provisions the OpenTelemetry extension, stages the config, and a human reloads. Method-level spans, the whole waterfall. If anything on the box was going to scribble on a zval mid-request, our extension was the prime suspect.

So we ran the matrix. Newest build of the extension: crash. Upstream main with the latest over-free fixes: crash. WordPress hooks disabled: crash. Database, cache, and outbound-HTTP instrumentation each turned off: crash. Every instrumentation off: crash. Each result felt like progress and each one was a lie, because we were changing the passenger and the car kept crashing anyway.

Then we ran the control we should have run first. We unloaded the extension completely. Not disabled, not configured to no-op. Absent. The module was not in the process.

101 segfaults in eight minutes.

That single test detonated the entire investigation. If php-fpm crashes with our code nowhere in the address space, then every "still crashes with X off" result was evidence about nothing. There was a second, independent fault on that machine, and it had been contaminating every measurement we made. The timing had fooled us: the crashes started the same day we first enabled the tracing layer, so we had anchored on the extension and never questioned the ground it stood on.

The clean room

When your measurements are poisoned, you stop measuring on the poisoned bench.

We built a clean room: the exact same PHP package, the same WordPress, the same load profile, on a different physical machine. Then we ran the full product flow end to end. Provision the artifact, stage the config, apply the reload, generate traffic for hours.

2.9 million spans. Zero crashes.

Same software, byte for byte. Different hardware, no crash. At that point the conclusion writes itself, and it has nothing to do with PHP.

The bug was in the silicon

The crashing box ran a recent high-end Intel desktop part from the generation with the now well-documented instability, the Raptor Lake voltage problem that degrades some chips over their life. Elevated boost voltage, a shift in the minimum stable voltage, and eventually a core that computes the wrong answer under specific load and returns to normal when it cools off.

Here is the part worth keeping. A steady synthetic stress test passed clean on that box. stress-ng --verify was happy. The thing that reliably tripped the fault was the PHP workload: branchy, allocation-heavy, pointer-chasing, exactly the shape that turns a one-in-a-billion bad retire into a poisoned pointer you will later dereference. Our tracing layer made it worse only because it added work. It was an amplifier bolted onto sick silicon, not the thing doing the damage. A bit flips in a register, a pointer is now a small integer, and two hundred milliseconds later a class-constant read falls off a cliff.

The stress test asks "can you go fast for a long time." The real workload asks "can you be correct on this exact sequence of operations," and correctness is what a degrading core loses first.

What we took away

Rule out the machine before you accept "impossible." Memory corruption that survives disabling every plausible software cause, that leaves no writer when you write-protect the target region, that changes crash site while keeping the same root, is telling you the layer under your code is unreliable. We spent two nights inside PHP internals because we could not believe the hardware would betray us so quietly. It can. It does.

A workload can be a better hardware test than a stress test. If you run anything real on questionable silicon, use it as the detector. It exercises paths a synthetic loop never will.

And a few forensic notes, because getting a usable core out of php-fpm is its own small fight. The distribution debug symbols live in a separate package channel; install them or your backtrace is a wall of ??. The fpm pool drops the core size limit unless you raise it in the pool config, so a machine-wide ulimit is not enough. Cores from a busy PHP process run a couple hundred megabytes each, and a full disk writes a zero-byte core in perfect silence, which will waste an hour of your life if you let it. Check free space before you start, not after.

The best outcome of the whole exercise was the thing we did not ship. It would have been very easy to "fix" a hardware fault by adding a defensive guard to our extension, watch the crashes stop on the healthy boxes, and quietly poison every customer whose chip was fine. The bug report we almost filed against ourselves was wrong, and the code change we almost made would have been worse. Sometimes the win is the patch you talk yourself out of.