Tag: Incident Response
All the articles with the tag "Incident Response".
Selecting more than one tag in the sidebar shows only posts that have every selected tag.
- #SRE Postmortem#SRE#Postmortem
Lineth L2 Incident Management: Postmortem (5) - An L1 Deadlock the Engine API Made Permanent
The lab's local L1 stopped producing blocks for 3 h 14 min: after a one-slot re-org, the beacon node asked the execution client to build on a block's parent, and the Engine API says it must not. Settlement stopped behind it and the coordinator wedged. A Google SRE format postmortem with the logs, the one-RPC fix, and an SLI that hid the worst withdrawals.
9 min read - #SRE Postmortem#SRE#Postmortem
Lineth L2 Incident Management: Postmortem (4) - Alive but Dead: 12 Hours Without Grafana
A supervised kubectl port-forward lost its stream during a node memory stall and logged the same error 1,929 times over 12 hours. The process stayed alive, so its supervisor never restarted it. A Google SRE format postmortem on monitoring the monitoring, with the logs and a fix tested by injecting the failure.
5 min read - #SRE Postmortem#SRE#Postmortem
Lineth L2 Incident Management: Postmortem (3) - Started Before DNS: An RPC Node With No Peer for 49 Minutes
After a planned restart, the L2's public RPC node came up a few seconds before cluster DNS, failed to resolve its only peer once, and never tried again. It looked healthy for 49 minutes while no user transaction reached the sequencer. A Google SRE format postmortem with the logs and a one-probe fix.
7 min read - #SRE Postmortem#SRE#Postmortem
Lineth L2 Incident Management: Postmortem (2) - Two Counters, One Limit: Every Batch Proven Twice
Settlement fell 1,087 blocks behind. Every batch had been failing the prover's normal mode for days because the coordinator and the prover counted the same trace module differently, and the Mac under the cluster was swapping the prover's memory. A Google SRE format postmortem with the logs and the arithmetic.
9 min read - #SRE Postmortem#SRE#Postmortem
Lineth L2 Incident Management: Postmortem (1) - A BigInt Bug That Broke the Bridge's Self-Repair
Deposits from L1 to L2 went unclaimed for 1 h 46 min. The postman had a repair path for exactly this failure, and it crashed 1,125 times on a string that should have been a bigint. A blameless postmortem in the Google SRE format, with the logs, the fix, and the upstream PR.
9 min read - #AI#Kagent#SRE
Diagnosing OpenCTI with Kagent (6) - Drill 2: Invoice Failure After the SRE Upgrades
A blind rehearsal drill against the upgraded L402 monitoring. The pages fired in 2 minutes 21 seconds and the agent found the broken component, but it skipped a required tool and invented details. A blameless review of what happened.
12 min read - #AI#Kagent#Kubernetes
Diagnosing OpenCTI with Kagent (5) - Reviewing the Setup Against Google SRE
I reviewed the L402 payment gate's monitoring and on-call agent against the Google SRE Book and Workbook. Eight practices, what the review found, what changed, and the gaps that are still open.
9 min read - #AI#Kubernetes#Aperture
Diagnosing OpenCTI with Kagent (4) - Rehearsing Incidents with an LLM Agent
I rehearsed incidents against a Kagent on-call agent watching an Aperture L402 payment gate, then scored it against a playbook. The exercise found gaps in my own alerts too.
8 min read - #Incident Response#Vulnerability Response#Playbook
CISA Incident & Vulnerability Response Playbooks
Incident and Vulnerability Response Playbook Standard
15 min read