Tag: Postmortem
All the articles with the tag "Postmortem".
Selecting more than one tag in the sidebar shows only posts that have every selected tag.
- #SRE Postmortem#SRE#Postmortem
Lineth L2 Incident Management: Postmortem (5) - An L1 Deadlock the Engine API Made Permanent
The lab's local L1 stopped producing blocks for 3 h 14 min: after a one-slot re-org, the beacon node asked the execution client to build on a block's parent, and the Engine API says it must not. Settlement stopped behind it and the coordinator wedged. A Google SRE format postmortem with the logs, the one-RPC fix, and an SLI that hid the worst withdrawals.
9 min read - #SRE Postmortem#SRE#Postmortem
Lineth L2 Incident Management: Postmortem (4) - Alive but Dead: 12 Hours Without Grafana
A supervised kubectl port-forward lost its stream during a node memory stall and logged the same error 1,929 times over 12 hours. The process stayed alive, so its supervisor never restarted it. A Google SRE format postmortem on monitoring the monitoring, with the logs and a fix tested by injecting the failure.
5 min read - #SRE Postmortem#SRE#Postmortem
Lineth L2 Incident Management: Postmortem (3) - Started Before DNS: An RPC Node With No Peer for 49 Minutes
After a planned restart, the L2's public RPC node came up a few seconds before cluster DNS, failed to resolve its only peer once, and never tried again. It looked healthy for 49 minutes while no user transaction reached the sequencer. A Google SRE format postmortem with the logs and a one-probe fix.
7 min read - #SRE Postmortem#SRE#Postmortem
Lineth L2 Incident Management: Postmortem (2) - Two Counters, One Limit: Every Batch Proven Twice
Settlement fell 1,087 blocks behind. Every batch had been failing the prover's normal mode for days because the coordinator and the prover counted the same trace module differently, and the Mac under the cluster was swapping the prover's memory. A Google SRE format postmortem with the logs and the arithmetic.
9 min read - #SRE Postmortem#SRE#Postmortem
Lineth L2 Incident Management: Postmortem (1) - A BigInt Bug That Broke the Bridge's Self-Repair
Deposits from L1 to L2 went unclaimed for 1 h 46 min. The postman had a repair path for exactly this failure, and it crashed 1,125 times on a string that should have been a bigint. A blameless postmortem in the Google SRE format, with the logs, the fix, and the upstream PR.
9 min read