Lineth L2 Incident Management: Postmortem (3) - Started Before DNS: An RPC Node With No Peer for 49 Minutes

Part 3 of a series of postmortems from my Lineth L2 lab: one incident per failure type. Part 2 was capacity; this one is startup order and dependencies.

The lab’s L2 has two execution nodes. The sequencer builds blocks. The RPC node is what users and services talk to: it accepts transactions and gossips them to the sequencer over P2P. Its only peer is the sequencer, configured as a static peer by DNS name. Format: the Google SRE postmortem template, blameless.

Summary

Impact

Error budget. tx-inclusion-60s (99 % of user transactions included within 60 s) went from 96 % to 90 % of its 28-day budget remaining, and tx-inclusion-5s from 91 % to 90 %. Both understate it: the SLI is built from user-transaction traces that finished. A transaction the generator gave up on never finishes and is never counted, so the worst part of the outage was invisible to the SLO (the same blind spot as in Parts 1 and 2).

Root causes and trigger

Trigger. The Docker VM was restarted at 04:17 to give the Mac memory back (Part 2). On a full node restart every pod starts at once, and cluster DNS answers only some seconds later. Synthetic clients showed the window:

l2-load | urllib3.exceptions.NameResolutionError: HTTPConnection(host='l2-node-besu', port=8545):
          Failed to resolve 'l2-node-besu' ([Errno -3] Temporary failure in name resolution)

They retry per request and recovered on their own. The RPC node did not:

2026-10-13 04:17:48.766+0000 | main | INFO  | Besu | Connecting to 1 static nodes.

That is the last peer-related line until the pod was deleted 38 minutes later. The static peer is enode://…@sequencer.lineth-l2.svc.cluster.local:30303; the name did not resolve yet.

Root cause 1: resolve once, drop silently. Besu resolves a static peer’s hostname when it starts. When that fails it drops the peer without a warning and never retries. Discovery could not help either:

WARN | Besu | --discovery-mode=BOTH but no ENR (enr:) bootnodes provided; DiscV5 will start with no bootstrap peers.

Root cause 2: the probe checked the wrong thing. The startup probe was a TCP check on the RPC port. A node with zero peers has an open RPC port, accepts transactions, answers eth_blockNumber, and is “Ready”, so Kubernetes saw nothing wrong. Transactions sent to it went into its own pool and nowhere else. Senders saw failures like:

04:25:20 FAILED: l2-send: attempt 1/3 nonce=955 failed: Error: server returned an error response:
  error code -32000: Replacement transaction underpriced

(the previous attempt with the same nonce was still sitting in the peer-less node’s pool).

Contributing. The manifest comment said the static peer was “at the sequencer Service ClusterIP”, but the config held the DNS name. The risk looked handled.

Why only this pod. Every other component checked: the coordinator, the postman and the synthetic users resolve names per request and retried; the sequencer has no static peers; the remote signer already waits for its dependency (an init container added after an earlier incident with the same “started before its dependency” shape).

Resolution

1. Mitigation (04:56): confirmed the name now resolves, deleted the pod. It peered on start; the pool drained from 67 to 5 and the synthetic users logged ok again by ~05:06.

2. Fix (05:26): a startup probe on Besu’s built-in readiness endpoint, which answers 200 only with at least the given number of peers:

   # Started = has a peer (Besu /readiness, minPeers=1), not just an open port: a node that came up without
   # its static peer is restarted after 2 min instead of running with 0 peers.
   startupProbe:
     httpGet: {path: "/readiness?minPeers=1", port: rpc}
     periodSeconds: 5
     failureThreshold: 24

   Checked before relying on it: /readiness?minPeers=1 → 200, /readiness?minPeers=2 → 503 on a node with one peer. The new pod was Ready, with its peer, 6 seconds after start.

A wait-for-DNS init container was considered and dropped. It covers only DNS; the probe covers any reason for “no peer” with one mechanism, and costs at most 2 minutes after a restart.

Detection

The alerts were right and early and reached no person. The human found it while measuring something else.

Lessons learned

What went well

• Before choosing a fix, every other component was checked for the same “resolve once” pattern from its logs.
• The fix uses an endpoint the node already has and tests the property that matters (has a peer), not a proxy for it.

What went wrong

• A planned restart had no post-restart check of user-facing workflows. It was called done when the pods were Running.
• This was the second incident in two days caused by a pod starting before something it needed.
• The critical alert fired in 3 minutes and was seen after 37.

Where we got lucky

• Block production and finalization do not depend on the RPC node, so the damage stayed on the user side.

Action items

Timeline (UTC, 2026-10-13)

Takeaways

• “Started” should mean “can do its job”. A port check proves a process is listening, not that it is useful. If the software has a readiness endpoint that knows (here, peer count), probe that.
• Anything resolved once at start is a restart-order bug waiting for a full restart. Rolling restarts hide it; a node reboot finds it.
• A restart is a change. Verify the user-facing paths after it, not just the pod status; this restart also caused the next incident.