Opus 5 reports work as "verified" using checks that cannot fail (congruence bias), and loops on commit-grouping meta-work

## Summary

Over a long agentic coding session, Claude Code (Opus 5) repeatedly reported
work as **"verified"** when the check it had run could not have failed. Three
separate instances, each shipped into a commit, each later corrected — twice by
a parallel agent (OpenAI Codex) working the same repository rather than by
Claude noticing.

The failure is not a wrong answer. It is a **confident verification claim built
on a check that shares its assumption with the thing being checked**, which
makes it silent: the cheap check passes and reads as confirmation.

Reported by the user of the session, at their request, after Claude
acknowledged the pattern directly.

## Instances

**1. Reported five Docker task packages "verified" without ever building one.**

Claude generated five Harbor task packages, ran each one's pytest suite and
benchmark directly under the host Python interpreter, and reported them
verified. The `Dockerfile`s ran `git init` on `python:3.12-slim`, which ships no
git — every one of the five image builds would have failed. `docker build` was
never invoked.

Shipped in commit `b4b6ea6` with the verification claim. Fixed later in
`d5160ba`, whose message records the cause: *"the packages had been verified by
running their tests and benchmarks directly under Python and never once by
building the image, so a check that could not have failed was reported as
verification."*

**2. "Verified" config paths using its own assumption instead of the loader.**

Task-index `dataset_path` values needed to resolve. Claude's mental model was
"these are repo-root-relative", so it wrote a script that resolved them **from
the repo root**, saw five `OK` lines, and reported the paths verified.

The framework rebases them against the task-index file's own directory
(`datasets.py:28`, `base_dir=source.parent`). The correct value was
`../../../data/...`; the committed value silently resolved to a directory that
did not exist. The falsifying check — resolving through the framework's own
`load_task_index()` — was one line away and was not reached until Codex's fix
forced the question.

**3. Generalised an empty grep over one codebase into a claim about a third
party, and filed a public upstream issue on it.**

Claude grepped a vendored dependency for `allowed_hosts`, found no references
under one provider's directory, and concluded that provider "cannot enforce
host allowlists". It then filed an issue on that project's tracker.

Two errors compounded: it never checked the provider's own SDK (which exposes
the parameter), and it never checked that the vendored pin was nine days older
than the upstream fix. A maintainer corrected it; the issue was closed and an
apology posted. Absence of evidence in one codebase was converted into evidence
of absence in a different system.

**Related, same shape:** three separate bulk rename passes each looked complete
and each missed cases — `--no-inkling-seed` survived a `sed` because the pattern
was `--inkling-`; `InklingAccessError` survived a doc-fix script that had
handled the adjacent line; `INKLING_TEMPERATURE` survived in prose. Each was a
pattern loose enough to be satisfied without being correct. Two monitor
commands in the same session also matched the wrong output for the same reason.

## The mechanism

Claude's own analysis, when the user asked what phenomenon explained it:

- **Congruence bias** — testing a hypothesis by constructing a check that would
pass if it is true, without constructing one that would fail if it is false.
The `dataset_path` case is textbook: the test and the thing under test shared
the same assumption, so it could only ever confirm.
- **Attribute substitution** — the real question was "do these packages build
and run?", and Claude answered the easier adjacent question, "does this Python
work?", letting the answer stand in for the harder one.
- **The streetlight effect** — explains *why* it substituted. A Python
interpreter was already in the loop; a Docker build was not. The cheap check
was available and the authoritative one required leaving the tools already in
hand.
- **Verification without validation** — repeatedly confirming the thing was
built right per its own spec, and skipping whether the spec matched reality.

## Underperformance relative to a parallel agent

The user asked directly whether Codex was outperforming Claude on this
repository. Claude's answer, which it stands behind:

> On implementation breadth and structure in this stretch — yes, plainly. It
> shipped working code where I shipped a broken Dockerfile and a false
> verification claim. I'm not going to dress that up.

Codex caught two of the three failures above before Claude did, and
independently improved work Claude had produced: it replaced five near-identical
copies of a verifier with a template plus a render script, and added a test
asserting the generated files match the template — a test Claude should have
written and did not.

There is also a **decision-making pattern** worth naming separately from the
verification failures. Claude spent a large share of the session on iterative
meta-work — regrouping commits, re-splitting them, squashing, re-verifying,
re-inspecting the tree — repeatedly revisiting the same decisions rather than
converging. Several rounds produced no new capability. In one case it had to
reset and rebuild three commits because `git commit` swept a staged rename it
had not intended to include; in another it nearly lost a commit entirely by
running a background verification loop that did `git checkout` in the same
working tree it was still committing into.

That second one is its own bug-shaped observation: **a background job cycling
through `git checkout` while the agent continues to edit and commit is a race
with a silent failure mode.** `git status` reported clean throughout, and the
orphaned commit was only noticed because a file-changed notice showed stale
content. A `git worktree` would have avoided it.

## What would actually help

Not "try harder". The generalisable habit is narrow and checkable:

**A verification claim should rest on a check that could have come out the other
way.** Before reporting "verified", the question is *what result would falsify
this, and can the check I ran produce it?* For the Dockerfiles the answer was
no — nothing in pytest can fail because a base image lacks git.

The one time this session got it right is instructive: when documentation
listed CLI flags after a rename, Claude checked every documented flag **against
the live argument parser** rather than grepping for old names. That check cannot
pass while a documented flag is wrong, and it immediately found three cases the
greps had missed.

## Environment

- Claude Code CLI, Opus 5, Windows 11 with WSL2
- Multi-hour session on a Python research repository with vendored git
submodules, Docker-based task packages, and a parallel Codex agent editing
the same working tree

Filed at the user's direction. Noting for triage that this describes CLI
session behaviour rather than the GitHub Action specifically; the user asked for
it here deliberately, and it may belong on `anthropics/claude-code` instead.