escalation-gate: from a GitHub issue to a checked root cause and the PR that fixes it

HackerRank’s hiring-agent scores résumés with an LLM. Issue #424 says the same résumé gets different scores on different runs. I scored one résumé 30 times: the spread was 6 points at the default temperature and 3 at temperature 0. The repo had 192 open pull requests. Some of them claimed to fix scoring consistency. Which one actually fixes this bug?

Answering that by hand took me about 1.5 hours: trace the cause through four files, then check every open PR against it. escalation-gate does it in one command for $0.41, and it hands an engineer one page they can act on in five minutes.

Six stages, one command

uv run python -m gate.cli --repo interviewstreet/hiring-agent --issue 424

• 01Collect — open issues, open PRs and every PR’s diff, from the GitHub CLI into Postgres.
• 02Investigate — Claude Code, headless and read-only (Read, Grep, Glob), finds the cause and cites each claim as file:line:quote, plus the chain from cause to symptom.
• 03Check — every citation is matched against the source. A miss gets one retry in the same session with the exact failures; a second miss goes to a human.
• 04Shortlist — SQL over diff hunks keeps only the PRs that change a cited line or mention the issue. No model, no cost.
• 05Judge — one model call with no tools reads the shortlisted diffs and labels each fixes, partly or no, with one reason.
• 06Packet — citations re-checked, git merge-tree per candidate PR, one HTML page.

Check the citations, label the reasoning

I treat model output as untrusted by default. A citation is cheap to verify: open the file, go to the line, look for the quote within five lines either side, whitespace ignored. So every citation is verified, mechanically. The explanation that links the citations is not. The packet says which is which, line by line.

Merges are the same. "This PR merges clean" is a dry-run git merge-tree against the pinned commit, not an opinion. "This PR fixes the bug" is judged from the diff and labelled that way, because nothing was run.

The shortlist needs no AI

A diff hunk header like @@ -240,10 says "this change replaces lines 240–249". If a cited line falls inside that range, the PR touches the bug. The cited lines are already verified, so the filter adds no new guess. On #424 it cut 192 open PRs to 25, and the three that matter were all still in.

It caught a mistake in my own review

I wrote the #424 packet by hand first, as the target the tool had to match. My review said #415 only partly fixes the bug. The tool said it fixes it fully. I opened the diff: #415 adds resolve_selected_projects, which rebuilds every selected project from the fetched data. The tool was right. It also found #242, a partial fix I had missed.

And where it was wrong

On a second bug, the investigation said GitHub’s /users/{user}/repos?type=all never returns repos owned by an org or another user. That is a claim about GitHub, not about the code, so the checker can’t verify it, and the packet flags it as model reasoning. I tested it against the live API: false. Those repos do come back. The real bug was one line: github.py looks up contributors with the candidate’s username as the repo owner. On a well-known public profile that mislabels 25 of 99 repos as solo projects with zero commits. Fixing it is a two-line change — I filed it upstream as issue #479 with the fix in PR #480.

That is the point of labelling. A tool that hides the difference between checked and guessed is worse than no tool.

Numbers

• 01#424: 6/6 citations verified · 192 → 25 PRs shortlisted · #425 recommended, merges clean · $0.11 to investigate + $0.30 to judge, on Sonnet.
• 02Python, Postgres, the GitHub CLI and Claude Code headless (--json-schema, --session-id, stream-json logs). 54 pytest tests, none of which call the model.
• 03Limits: tested end to end on two unrelated repos so far (hiring-agent and spark-stratifier), and runs vary. Next is V2 — apply the recommended PR to a throwaway copy, rerun the reproduction, and put the measured result in the packet.

Code: github.com/rdeepak-711/escalation-gate. Live boards, packets and two short videos: escalation-gate-rdeepak.vercel.app. Upstream fix: github.com/interviewstreet/hiring-agent/pull/480.