NVIDIA / NVIDIA/NemoClaw

Fix repeated PR Review Advisor evidence-read failures

Open
#11,392 0 comments 0 reactions 0 assignees View on GitHub
area: ci
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Investigation summary

- PR Review Advisor run 34439476165 evaluated PR #11366 at candidate `70c851bdb6cece2cb782a973ba0fb2ade587e1c6`.
- Attempt 1 failed three specialists because the advisor turn omitted the required evidence read.
- A maintainer-approved full rerun recovered Documentation Drift and Customer Value, but Verification Evidence failed the same way again in attempt 2.
- The failed job uploaded no review artifact; eight other specialists completed successfully.
- The repeated failure required a maintainer deferral and prevented the Advisor link publication step.

## Description

The PR Review Advisor specialist lifecycle can finish without reading its supplied evidence. The job then fails validation and produces no specialist artifact. A full workflow rerun does not reliably recover the missing review.

Expected: every scheduled specialist reads the required evidence and publishes a review artifact, or the workflow reports a narrow transient result that repository policy can retry safely.

Actual: Verification Evidence failed in both attempts before producing `pr-review-specialist-verification-mistake-proofing-1` or `pr-review-specialist-verification-mistake-proofing-2`.

## Reproduction steps

1. Run the full PR Review Advisor for PR #11366 at candidate `70c851bdb6cece2cb782a973ba0fb2ade587e1c6`.
2. Observe Verification Evidence fail during the advisor specialist lifecycle.
3. Rerun the complete workflow without changing the candidate.
4. Observe the same specialist fail again with no review artifact.

## Environment

- Repository: NVIDIA/NemoClaw
- Workflow: Automation / PR Review Advisor
- Run: https://github.com/NVIDIA/NemoClaw/actions/runs/34439476165
- Attempt 1 job: https://github.com/NVIDIA/NemoClaw/actions/runs/34439476165/job/102751485791
- Attempt 2 job: https://github.com/NVIDIA/NemoClaw/actions/runs/34439476165/job/102753691886

## Debug output

Both jobs report that the advisor specialist lifecycle did not complete. The artifact upload then reports that no files exist under `artifacts/pr-review-specialist-verification-mistake-proofing/`.

## Checklist

- [x] I confirmed this bug is reproducible.
- [x] I searched existing issues and this is not a duplicate.

Contributor guide

Open the contributing guide

Research direction

Start with the Automation / PR Review Advisor workflow and inspect the specialist lifecycle and artifact-upload logs for run 34439476165, especially the Verification Evidence failures. Reproduce with PR #11366 and candidate 70c851bdb6cece2cb782a973ba0fb2ade587e1c6; done means each scheduled specialist reads its evidence and publishes an artifact, or reports a narrowly retryable transient result.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
ai-infra-agents, ci-cd
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.