awslabs / awslabs/cli-agent-orchestrator

[Feat] Add autonomous headless PR reviewer to CI

Open
#464 4 comments 0 reactions 1 assignee Claimed by @gutosantos82 View on GitHub
enhancement
Dominant language
Python
Stars
1.3k
Forks
267
Avg merge
1d 23h
Merged PRs (30d)
70

Description

## Overview

Code review is becoming a delivery bottleneck for CAO. Add a standalone GitHub Actions job that launches an autonomous coding agent in headless mode to provide an evidence-backed first review of trusted pull requests.

Kiro CLI is the initial engine. The design should keep a small provider adapter so other headless coding CLIs can later be evaluated against the same review contract. This feature must not depend on CAO sessions, workflows, or the CAO application runtime.

## User Stories

- As a PR author, I want early risk and defect feedback so I can address material issues before human review.
- As a maintainer, I want findings prioritized by severity so I can focus review effort on the highest-risk changes.
- As a maintainer, I want high-confidence material defects to block merge without allowing low-confidence model output to create noise.

## Acceptance Criteria

- A GitHub Actions workflow runs on trusted same-repository PRs when they are opened, reopened, marked ready for review, or updated.
- A stale review is cancelled or superseded when the PR head SHA changes.
- The workflow launches Kiro CLI in headless mode with a dedicated read-only reviewer profile.
- The reviewer can inspect repository files and diffs but cannot modify files or execute PR-controlled code.
- Test and build evidence is consumed from separate normal CI jobs.
- The review considers correctness, regressions, edge cases, security, compatibility, test quality, performance, operations, and maintainability.
- The report contains an overall risk level with concrete reasons and classifies findings as `Must Fix`, `Should Fix`, or `Good To Fix`.
- Every finding includes a path and line, failure scenario, impact, evidence, confidence, and suggested remediation direction.
- Security findings also identify the relevant trust boundary, attacker capability, and exploit path.
- Agent output follows a machine-validated contract; malformed output, stale SHAs, invalid line references, or unsupported severity values are rejected.
- The workflow updates one persistent GitHub Check or summary instead of creating repeated comment noise.
- The check fails and blocks merge only for validated, high-confidence `Must Fix` findings. Risk and lower-tier findings remain advisory.
- CLI version, model selection, review rubric, timeout, and credential handling are pinned or auditable.
- The reviewer first runs in shadow mode against approximately 20-30 historical or live PRs; blocking mode is enabled only after maintainers validate its precision.

## Proposed Solution

- Add a dedicated AI review workflow under `.github/workflows/`.
- Add a versioned reviewer profile and review rubric for Kiro CLI.
- Add a thin runner/adapter responsible for CLI installation, authentication, invocation, timeout, and output capture.
- Validate the structured report locally before publishing the GitHub Check conclusion.
- Keep the engine contract provider-neutral enough to evaluate Codex CLI, Claude Code, or Gemini CLI later.
- Evaluate a fresh verification pass for candidate `Must Fix` findings before enabling the required check.

## Security Boundary

The credential-bearing reviewer job must not execute PR-controlled code. Do not use a privileged `pull_request_target` workflow that checks out and executes an untrusted PR head. Model credentials must not be printed, passed to test processes, or exposed in review output.

## Out of Scope

- Public fork PR support in V1
- Automatic code fixes
- Arbitrary command or test execution by the reviewer
- Multi-agent review panels

## Alternatives Considered

- **Direct model API:** simpler structured output, but preselected context can miss cross-file evidence that an autonomous repository reviewer should discover.
- **Managed PR-review service:** faster adoption, but less control over CAO's risk taxonomy, evidence requirements, blocking policy, and model choice.
- **Multi-reviewer panel:** potentially broader coverage, but too much cost, latency, and duplication before a single reviewer is calibrated.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.