unic / unic/unic-agents-plugins

bug(unic-pr-review): intent-check merger discards all verdicts when an item has a `note` — Intent Check silently renders as all-unaddressed

Open
#274 0 comments 0 reactions 1 assignee View on GitHub

@orioltf is already working on this.

Since Aug 11, 2026.

app:unic-pr-review bug javascript needs-triage p1
Dominant language
JavaScript
Stars
1
Forks
0
Avg merge
16h 43m
Merged PRs (30d)
19

Description

Summary

intent-check-merger.mjs short-circuits on any skeleton item that carries a note, returning it with verdicts untouched. Because the Intent Checker emits note on ordinary items (to record AC provenance), a fully-assessed Intent Check is silently discarded and every AC renders as unaddressed.

This reproduces the exact symptom of the closed #160 ("Intent Check verdicts are static (always unaddressed)") through a different mechanism.

Root cause

apps/claude-code/unic-pr-review/scripts/lib/intent-check-merger.mjs:102

const items = skeleton.map((skeletonItem) => {
    if (skeletonItem.note !== undefined) {
        return skeletonItem          // <-- verdicts never overlaid
    }
    ...

The module header documents this as the ADR-0004 hard-stop passthrough:

* items (ADR-0004 hard-stop signal) pass through verbatim, verdicts untouched.

…and agents/intent-checker.md:59 documents note only for the not-found soft-error case:

{ "id": "<item.id>", "title": "Work Item <item.id>", "verdicts": {}, "note": "Work Item not found." }

But note is a free-text field on a schema handed to an LLM agent, and the natural reading is "annotate this item". In practice the Intent Checker uses it for legitimate provenance on healthy items. From a real run against ADO PR 5647 (4 linked work items, all reachable, all assessed):

"note": "No AcceptanceCriteria field on the work item; ACs enumerated from Feature FPTD-01-b on Confluence page 603390377. AC 2, AC 3 and AC 4 are struck through in the work item's ToDo list and are descoped."

All four items carried a note of this kind. So note is overloaded: it means both "this item is a hard-stop, do not touch" and "here is context about this item".

Reproduction

SKELETON_JSON='[{"id":"1","title":"T","verdicts":{"AC 1":"unaddressed"},"note":"ACs enumerated from Confluence page 12345."}]' \
ASSESSED_JSON='[{"id":"1","verdicts":{"AC 1":"addressed"}}]' \
node apps/claude-code/unic-pr-review/scripts/lib/intent-check-merger.mjs

Actual

{"items":[{"id":"1","title":"T","verdicts":{"AC 1":"unaddressed"},"note":"..."}],
 "diagnostics":{"assessedReceived":1,"applied":0,"droppedElements":0,"rejectedVerdicts":0,"unmatchedItems":0}}

ExpectedAC 1 overlaid as addressed, applied: 1.

Dropping the note key from the same input yields applied: 1, confirming note is the sole trigger.

Impact

Silent and total, and it degrades in the most misleading possible direction — a reviewer sees a plausible, fully-populated Intent Check block asserting that the PR addresses none of its acceptance criteria.

In the PR 5647 run this suppressed all 41 verdicts across 4 work items. The correct output was 25 addressed / 11 partially addressed / 5 unaddressed. The first render claimed 0 / 0 / 41. It was caught only because applied: 0 looked wrong against an Assessor that had visibly returned verdicts; nothing in the rendered summary flagged it.

Note the diagnostics cannot distinguish this failure from a legitimate one: droppedElements, rejectedVerdicts and unmatchedItems are all 0, exactly as in a healthy run where the Assessor genuinely returned nothing. So the maintainer-facing stderr drift classes in review-pr.md Step 8 item 4 do not fire either.

The module's own header names this risk precisely:

a wholesale silent fallback to unaddressed is indistinguishable, to the Reviewer, from a genuine "the diff doesn't address this"

There is a partial safety net — review-pr.md Step 8 item 3 requires the orchestrator to set unassessedIntentCheck: true when applied === 0, which raises a reviewer-facing Notice. It depends on the orchestrator checking applied, and it reports the wrong cause even when it fires (it says the Assessor produced no valid verdicts; the Assessor was fine).

Suggested fix

The overload is the bug; note should not be a control signal. Options, roughly in order of preference:

  1. Use an explicit field for the hard-stop. Have the Intent Checker set something like "hardStop": true (or "unreachable": true) on items that must pass through untouched, and key the short-circuit on that. note then goes back to being free-text documentation. Requires a matching change in agents/intent-checker.md.
  2. Key the short-circuit on the actual hard-stop shapeObject.keys(skeletonItem.verdicts).length === 0. An item with no ACs to assess has nothing to overlay, which is the real invariant behind the not-found case, and it does not care what note says.
  3. At minimum, make it loud. Count short-circuited items in diagnostics (e.g. passthroughItems) so applied: 0 with passthroughItems: 4 is distinguishable from a genuinely empty Assessor response, and extend the Step 8 stderr drift classes to name it.

(2) is the smallest change and fixes the reported case; (1) is the one that stops this recurring.

A regression test with a note on a healthy, fully-assessed item would have caught this — worth adding alongside whichever fix lands.

Environment

Plugin unic-pr-review 2.1.12
Repo commit d7dd452
Node v22
Observed in /unic-pr-review:review-pr against ADO PR (re-review, 4 linked work items, 41 ACs)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.