skills/regression-bisect/SKILL.md (4191 bytes)
1 --- 2 name: regression-bisect 3 description: Find the exact commit that introduced a regression using git bisect. Use when something used to work and now does not, when a test that previously passed now fails, when a performance or memory regression appeared between releases, or when the user says "this broke", "worked last week", or "which commit caused this". Prefer this over reading diffs or reasoning about likely causes. 4 --- 5 6 # Regression bisect 7 8 Do not reason about which commit "probably" caused a regression. Bisect finds it 9 in log2(n) steps with certainty. A 1000-commit range resolves in ~10 builds. 10 11 ## Step 1: Build a reproducer that exits with a status code 12 13 This is the whole job. Everything else is mechanical. 14 15 The reproducer must exit **0 when the code is good** and **non-zero when bad**. 16 Make it as fast and as narrow as possible — it will run once per bisect step. 17 18 ```bash 19 #!/usr/bin/env bash 20 # /tmp/bisect-test.sh 21 set -euo pipefail 22 23 # Build. If the build itself fails, this commit cannot be judged: exit 125. 24 make -j"$(nproc)" >/dev/null 2>&1 || exit 125 25 26 # The actual check. Narrow it to the single failing case. 27 pytest tests/test_parser.py::test_nested_arrays -q 28 ``` 29 30 Exit code contract for `git bisect run`: 31 32 | Code | Meaning | 33 |---|---| 34 | 0 | good | 35 | 1–124, 126, 127 | bad | 36 | 125 | untestable, skip this commit | 37 | 128+ | abort the bisect | 38 39 **Store the script outside the repository** (`/tmp`, or a path in 40 `.git/info/exclude`). Bisect checks out old trees; a script inside the worktree 41 will be overwritten or deleted mid-run. 42 43 `chmod +x` it. Verify it manually on a known-bad commit (expect non-zero) and a 44 known-good one (expect zero) before starting. A miscalibrated script silently 45 produces a confident wrong answer. 46 47 ## Step 2: Establish the bounds 48 49 ```bash 50 git status # must be clean; stash or commit first 51 git bisect start 52 git bisect bad # current HEAD is broken 53 git bisect good v1.4.0 # last known-good tag or commit 54 ``` 55 56 If no known-good point is known, walk backwards until the reproducer passes: 57 58 ```bash 59 git checkout HEAD~50 && /tmp/bisect-test.sh; echo "exit=$?" 60 ``` 61 62 Double the step each time until it passes, then use that as the good bound. 63 64 ## Step 3: Run it 65 66 ```bash 67 git bisect run /tmp/bisect-test.sh 68 ``` 69 70 Report the `<sha> is the first bad commit` output verbatim, plus 71 `git show --stat <sha>`. 72 73 Then, always: 74 75 ```bash 76 git bisect reset 77 ``` 78 79 Leaving a repository in a bisecting state is a trap for the next session. 80 81 ## Variants 82 83 **Performance regression** — bisect on a threshold, not on pass/fail: 84 85 ```bash 86 ns=$(hyperfine --style none --export-json /dev/stdout ./bench \ 87 | jq '.results[0].mean') 88 awk -v t="$ns" 'BEGIN { exit (t > 0.250) ? 1 : 0 }' 89 ``` 90 91 Pick the threshold to sit clearly between the known-good and known-bad 92 measurements, not at the midpoint of noise. Run the benchmark on both bounds 93 first to see the spread. 94 95 **Flaky failure** — the reproducer must loop, or bisect will mark good commits 96 bad and give a wrong answer: 97 98 ```bash 99 for i in $(seq 1 30); do ./run-test || exit 1; done 100 ``` 101 102 If the failure rate is below ~10%, raise the iteration count until a known-bad 103 commit fails reliably. If it cannot be made reliable, bisect is the wrong tool — 104 use `dynamic-analysis` (TSan for races, ASan for use-after-free) instead. 105 106 **Noisy history** — skip merge commits or vendored-code churn: 107 108 ```bash 109 git bisect start --first-parent # follow mainline only 110 git bisect skip $(git rev-list --merges good..bad) 111 ``` 112 113 **Bug is in a dependency, not your code** — bisect the dependency's repo with 114 your reproducer pinned against it, or bisect your lockfile history. 115 116 ## After the culprit is found 117 118 1. Read the diff. Explain the mechanism — *why* that change causes this 119 failure. If the mechanism is not clear, the bisect result may be a 120 coincidence of a flaky reproducer; re-verify by checking out the parent 121 commit and confirming it passes. 122 2. Write a regression test that fails on the bad commit and passes on its 123 parent. Show both runs. 124 3. Then fix it. 125 126 Do not revert without understanding the mechanism. The bad commit may have been 127 fixing something else.