Triage a failing build

---name: failing-builddescription: "Triage a red CI run: find the first real failure, attribute it to a change, and say whether the code is wrong or the runner was."metadata:  title: "Triage a failing build"  category: engineering  tags: [ci, debugging]--- ## When to use this A build went red and somebody has to decide whether to fix code, fix a test, orfix the pipeline. ## How 1. Find the FIRST failure in the log, not the last. A failing run usually ends   in a cascade of steps that failed because an earlier one did, so the error   nearest the bottom is the least informative line in the file.2. Read the assertion rather than the summary. "1 failed" says nothing; the   expected-versus-actual says what the code believes about itself.3. Decide which of three things happened before proposing anything, because   they have different owners:   - the change under test is wrong;   - the change is right and the test is stale;   - nothing is wrong and the environment failed -- a refused download, a full     disk, an unset secret, a port already bound.4. Attribute it. Find the last green run and name the change in between that   touched what broke. If the same job has failed on unrelated changes, it is   flaky, and then the test is the defect.5. Report the cause and the smallest fix for it. Never re-run a job to get it   green without saying why it was red: a re-run that passes has not answered   the question, it has only stopped asking it.