The retry loop that sent a thousand alerts

A network hiccup dropped a handful of agents for about a minute. They reconnected, recovered, and the whole event deserved one line in a timeline. What actually happened over the next two hours was 1,160 incident state transitions and hundreds of chat digests, all describing that same minute, over and over, like a scratched record.

No component was down. Every component was doing its job. The loop lived in the seams.

Anatomy of a storm

The alert pipeline has a producer and a consumer. The producer detects state transitions and pushes them in batches; if the consumer is unreachable, batches go to a write-ahead log and get replayed. Boring, correct design.

The consumer, on receiving a batch, processed it fully before acknowledging: create incidents, update timelines, and send notifications, including email over synchronous SMTP. On a normal day that fits inside the producer's timeout. On this day, the blip produced a batch of ten transitions at once, the notification fan-out for ten incidents took longer than the timeout, and the producer never saw the ack.

So the producer did what unacknowledged delivery means: kept the batch in the WAL and replayed it. Every thirty seconds. The consumer, perfectly healthy, processed every replay as new truth: reopen, resolve, notify, repeat. Each cycle took just long enough to miss the ack again. A stable orbit.

The messenger hid the evidence

The most instructive part is why it ran for two hours before a human understood. The chat platform throttles bots that flood, so the hundreds of digests were being queued on their side and dribbled out slowly. From the operator's chair, messages were arriving at a calm, steady pace. It looked like a backlog draining, not a loop running.

"Notifications are still coming in" is evidence about the messenger's queue, not about your system's present. The ground truth was in the incident event ledger, where the same fingerprint reopened and resolved twenty times an hour, plain as day, the whole time. We looked at the inbox because the inbox is what beeps. The ledger is what knows.

Three fixes, one on each seam

Ack-first on the consumer: acknowledge receipt the moment the batch is durably accepted, close the connection politely, then do the slow work of delivering notifications. An ack means "I have it," not "I have finished feeling all my feelings about it." Mail moved to a detached worker while it was at it, so nothing interactive ever waits on SMTP again.

Coalescing on the producer: WAL replay now collapses entries to the last state per fingerprint before sending. Replaying history is pointless when only the final state matters; a resolve that follows a reopen cancels it in the log, not in the consumer's face. And failed drains log loudly instead of retrying in silence.

A cooldown on the mouth: no matter what the state machine does, one incident does not notify humans more than once per half hour. Even if some future loop finds a new seam, it will burn quietly in the ledger instead of spending everyone's attention.

The shape to remember

Every piece here was individually defensible. Durable delivery with replay: correct. Full processing before ack: conservative, arguably correct. Messenger throttling: correct on their end. The storm existed only in the composition, and it was invisible in the place everyone watches.

Two habits fall out. First, decide explicitly what an ack means in every pipeline you own, and make it cheap, because everything behind a slow ack becomes a retry storm eventually. Second, when a notification channel misbehaves, go read the ledger it is fed from before you trust its pacing. The inbox tells you how fast the messenger talks. Only the ledger tells you how fast your system is lying.