chatgpt-review: pr mode returns a stale-head review after a push (restates prior findings, reports the old SHA) in long conversations

Summary

In pr mode, chatgpt-review.mjs sometimes returns a review of a stale PR head: the response restates findings from an earlier pass verbatim and self-reports reported_reviewed_sha as an older commit, even though (a) the fixes for those findings were pushed before the call, and (b) gh pr view <n> --json headRefOid confirmed the live head was the newer commit at call time. The independent fact-check stage downstream caught every instance (all stale findings were rejected as "already fixed at HEAD"), so no bad fix was applied — but each occurrence burns one of the session's three hard-capped pr-mode passes on a review of code that no longer exists.

Observed 3 times across 2 PRs in one /ship run (Altinity/altinity-oauth-helper), each time in a long-running ChatGPT conversation with many prior turns (plan authoring + prior review passes in the same thread), and each time immediately after a fix had been pushed between passes.

Reproduction (as observed)

1. Run a plan-author loop, then start pr mode with --seed-from-session <plan-session> on the PR. Pass 1 reviews the correct head and finds real issues.
2. Fix them, push, confirm gh pr view --json headRefOid shows the new head.
3. Re-invoke the same PR with --session <pr-session> (pass 2), or --seed-from-session <exhausted-session> to start a new pr session (pass 1 of session 2).
4. Response: the same findings as the prior pass, byte-for-byte in substance; reported_reviewed_sha = the pre-fix SHA. The published PR comment for that pass likewise cites the old SHA.

Instances (all in https://github.com/Altinity/altinity-oauth-helper):

Conversations: https://chatgpt.com/c/6a9098f3-a038-83eb-b5e2-852a4f1cd2bc (PR #21; sessions c16ce96d-caee-48ae-894b-7a77c59b0670, dd05fae0-8a2c-4c26-bce1-9f481fd10b46) and https://chatgpt.com/c/6a910d15-c1a0-83eb-aa16-d926786d07ad (PR #22; session 55cd505c-ece3-4547-9942-c614259682dd).

What made it go away

Appending an explicit instruction to the question file worked every time it was tried:

Your previous response reviewed a stale snapshot (self-reported reviewed_sha <old>; the live head is <new>, confirmed via gh pr view). Navigate to https://github.com/<owner>/<repo>/pull/<n>/files fresh and, BEFORE writing any finding, state the exact head SHA GitHub shows you and confirm it is <new>; if it differs, say so rather than proceeding.

The next pass then reported the correct SHA and produced genuinely new findings (PR #21 passes 5–6; PR #22 second-session pass 1).

Likely cause

scripts/lib/prompt.mjs:27 — the fix-review pass prompt reads:

This is fix-review pass N. Reuse your earlier analysis, fetch the new PR head, compare it with the previously reviewed SHA <previousSha>, reassess every earlier finding, and inspect the complete updated PR for regressions. Report both old and new exact SHAs.

It leads with "reuse your earlier analysis", and it tells the model the old SHA but never the expected new one — so there is nothing the model must verify against, and in a long conversation it can satisfy the prompt from memory ("new head" = whatever it last saw) and report the old SHA as "reviewed". The --seed-from-session first pass uses the fresh "Browse the canonical PR, clone or fetch the repository…" prompt (prompt.mjs:26) and still went stale once (PR #21 session 2 pass 1), so conversation memory dominates even the fresh-review wording when the thread is long. The fact that a plain-language "state the head SHA you see and confirm it equals <new>" instruction fixed it every time supports this.

Suggested fixes

1. Make the pass prompt state the expected head SHA and require confirmation. The CLI already knows (or can cheaply fetch via gh pr view --json headRefOid) the live head. Put it in every pr-mode prompt: "The PR head is <sha>. Re-open the Files view and confirm you are reviewing <sha> before writing findings; if GitHub shows a different SHA, stop and report it." This is exactly the manual workaround, automated.
2. Fail closed on SHA mismatch in the CLI. After the response, compare reported_reviewed_sha to the live headRefOid; if they differ, return a distinct status (e.g. stale_review) instead of completed, so callers (the /ship code-review-pass workflow) don't spend fact-check agents and a pass slot on it. Ideally the CLI retries once with the re-navigate instruction before giving up.
3. Don't count a stale review against the 3-pass cap (or at least surface it), since the cap exists to bound real review rounds. Note the interaction with the cap is what made this costly: on PR #22 a stale pass 2 plus pass 1's internal retries exhausted the session before pass 3 could run at all (session-cap-exhausted).
4. (Nice to have) In --seed-from-session starts, explicitly tell the model the conversation is being continued for a new review of the current head and that earlier findings may already be fixed.

Environment

• Skill at skills/chatgpt-review (this repo), consumed via ~/.claude/skills/chatgpt-review; Agent Chrome over CDP; Node 20+.
• Observed 2026-08-27/28 during /ship 19 in Altinity/altinity-oauth-helper; the downstream fact-check that caught it is altinity-oauth-helper/.claude/skills/ship/references/code-review-pass.workflow.mjs.

Beginne mit scripts/lib/prompt.mjs, insbesondere mit den Prompts für fresh-review und fix-review in den Zeilen 26–27, und untersuche anschließend, wie die CLI die PR head SHA abruft und verarbeitet. Verfolge den code-review-pass-Workflow in .claude/skills/ship/references/code-review-pass.workflow.mjs, um die Zählung der Durchläufe und den Umgang mit veralteten Reviews zu verstehen. Erledigt ist die Aufgabe, wenn der aktuelle Head explizit verifiziert wird und Abweichungen sichtbar gemacht werden, ohne einen gültigen Review-Durchlauf zu verbrauchen.