Lineth L2 Incident Management: Postmortem (5) - An L1 Deadlock the Engine API Made Permanent

Part 5 of a series of postmortems from my Lineth L2 lab: one incident per failure type. Part 4 was observability; this one is a distributed-system failure: two correct components that together could not make progress.

The L2 settles on a local L1 that is as small as an Ethereum chain can be: one execution client (Besu), one beacon node (Teku) and one validator client, with 1-second slots. The beacon node decides the head and, through the Engine API, tells the execution client which block to build on (engine_forkchoiceUpdated with payload attributes); the execution client returns a payload id, and the beacon node later fetches the built block. Format: the Google SRE postmortem template, blameless.

Summary

Impact

Error budget.

The withdrawal SLO should have taken the biggest hit; 57 withdrawals waited for hours. Its histogram explains why it did not. Over the two hours after recovery:

le=600 0 | le=1200 0 | le=1800 0 | le=3600 0 | le=7200 0 | le=+Inf 45

All 45 withdrawals took more than 2 hours. The SLI is good = bucket{le="3600"} over total = bucket{le="7200"}, so anything slower than 2 hours falls out of the denominator as well: the worst requests were not counted as bad, they were not counted at all. This is the third SLI blind spot in this series (Part 1: unfinished deposits, Part 2: lag that still moves), and the most direct: the total must be le="+Inf".

Root causes and trigger

Trigger. Block production had been running close to the edge: Slow Block Production warnings, several slots handled in the same second, 1–4 failed block proposals every two minutes in the half hour before. At 07:30:56–58 teku could not get a payload in time, and then re-orged its head back:

07:30:56.207 WARN  - No suitable payloadId for block production at slot 43094, requesting a new one to the EL
07:30:58.431 INFO  - Reorg Event *** New Head: 47b83d…edc (43093), Previous Head: 0512ef…363 (43095), Common Ancestor: 47b83d…

Besu had already imported the block of the orphaned slot (EL block 42491, 0x9ff3f…), so its canonical head stayed one block ahead of teku’s (EL 42490, 0x3bcbb…).

Root cause 1: a deadlock that the protocol makes permanent. From then on, every slot, teku sent forkchoiceUpdated with head = 42490 and asked for a payload. The Engine API specification allows the execution client to skip the update and says it must not start building a payload when the requested head is a valid ancestor of its canonical head. Besu answered VALID, with no payload id:

besu  07:31:30.502 | AbstractEngineForkchoiceUpdated | FCU(VALID) | head: 3bcbb.....29727 | safe: 32f91.....df01a | finalized: 35735.....18cee

and teku failed every proposal:

teku  07:30:58.471 ERROR - Failed to process request to URL http://teku:4000/eth/v3/validator/blocks/43096
java.lang.IllegalStateException: ExecutionPayloadContext is not provided for production of post-merge block at slot 43096

Failed proposals per two minutes (Loki): 1–4 before 07:30, 12 at 07:30, 70 at 07:32, then ~120: every slot for three hours. A new block would have broken the tie, but the only validator could not produce one. Each side was following its rules.

Root cause 2: no slack. One beacon node and one validator on 1-second slots, on a laptop shared with the rest of the lab. Slow proposals were routine; a re-org of this kind was a matter of time.

Root cause 3: the coordinator wedged and nothing restarted it. From ~07:38 the L2 coordinator’s /health and /metrics stopped answering; it stopped cutting batches (the last one ends at a block of 07:36). It was still hung at 10:50, five minutes after L1 recovered, so it was not just waiting for L1. It had readiness and startup probes only: Kubernetes marked it unready 1,150 times and never restarted it. Why it hung is unknown: no thread dump was taken before the restart.

Resolution

1. L1 (10:45). Confirmed that the head had not moved (eth_blockNumber = 0xa5fb), then pointed Besu’s head back at teku’s (Besu has the DEBUG RPC API enabled in this lab):

   before: {"jsonrpc":"2.0","id":1,"result":"0xa5fb"}
   head confirmed 42491 -> setHead 42490
   {"jsonrpc":"2.0","id":1,"result":"Success"}
   after 20s: {"jsonrpc":"2.0","id":1,"result":"0xa60d"}

   The dropped block was empty and already orphaned on the beacon side, so nothing was lost. Within 20 s L1 had produced 18 blocks; L1 finality caught up a few minutes later.

2. Coordinator (10:51). Restarted. It resumes from its database:

   10:52:48.588 | main | INFO | app | Resuming conflation from block=9913 inclusive blockTime=2026-10-13T07:36:23Z

3. Catch-up. The prover then had a backlog of 3,000+ blocks while the L2 kept adding ~17 blocks a minute. Pausing two synthetic-traffic generators (the L2 makes no empty blocks, so fewer transactions means fewer blocks to prove) cut L2 growth to ~7.7 blocks a minute. Closing apps on the Mac to relieve the memory pressure from Part 2 brought proofs back from 360–485 s to ~180–200 s per 80-block batch.

Detection

Every alert was right and on time. They went to a log and a console nobody was watching, as in every incident before this one. Fifteen minutes after the recovery, lab alerts were routed to Telegram, and the rest of the catch-up was followed there.

Lessons learned

What went well

• Cause found from logs and two RPC calls in about three minutes.
• The fix was the smallest possible (move one head pointer back past one empty, orphaned block) and the precondition was checked right before running it.
• During catch-up, measuring instead of guessing changed the plan twice: the backlog estimate was wrong (it ignored the L2’s own growth), and the slowdown came from the host, not the cluster.

What went wrong

• 3 h 11 min between a correct critical alert and a person: the eighth time in this lab.
• “Running but not working” again: a process with only a readiness probe can stay broken forever.
• The coordinator was restarted before a thread dump was taken; its root cause is lost.
• The withdrawal SLI dropped exactly the requests that mattered.

Where we got lucky

• The orphaned block was empty, so rewinding the head lost nothing.

Action items

Timeline (UTC, 2026-10-13)

Takeaways

• Two correct components can deadlock. Teku asked for something reasonable and Besu refused for a reason the specification gives. With one validator, nothing outside the pair could break the tie. Small test networks remove the redundancy that hides these states in production.
• Know your manual override before you need it. debug_setHead was enabled and known; that turned a possible chain reset into a one-line fix.
• Check the denominator. A latency SLI whose total is the largest finite bucket silently drops every request slower than that bucket: the outage disappears from the SLO in proportion to how bad it is.