aws / aws/aws-durable-execution-ci

Reduce long-tail latency in Claude PR reviews

Open
#4 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
1
Avg merge
14h 50m
Merged PRs (30d)
28

Description

## Problem

Claude PR review latency is highly variable and can hold the AI review workflow open long after the other reviewer has finished.

Recent examples from the consuming `aws-durable-execution-conformance-tests` repository:

- [Run 30653488287](https://github.com/aws/aws-durable-execution-conformance-tests/actions/runs/30653488287) (PR #43): the Claude model step took **15m55s** (`18:01:44` to `18:17:39`), while the parallel Codex model step took **5m16s**. Claude setup before model invocation took only about 10 seconds.
- [Run 30655388784](https://github.com/aws/aws-durable-execution-conformance-tests/actions/runs/30655388784) (PR #54): the Claude model step took **1m49s**.

This suggests a long-tail latency problem in model execution rather than workflow setup. Slow reviews delay feedback, consume runner capacity, and are more likely to be discarded when a newer commit cancels the in-progress run.

The current Claude job:

- Uses `us.anthropic.claude-opus-4-8`
- Allows up to 20 turns
- Has a 45-minute job timeout
- Repeats the full Claude action after any primary-attempt failure
- Supplies the complete PR diff and permits repository inspection with `Read`, `Grep`, and `Glob`

## Possible mitigations

These options can be evaluated independently; we do not necessarily need all of them.

1. **Measure the baseline.** Record model-step duration, changed-file count, diff size, turn count, retry reason, and token usage when available. Add a small job-summary table so latency can be compared without mining logs.
2. **Use a lower-latency model by default.** Compare the current Opus model with the current Claude Sonnet inference profile on a representative set of PRs. Consider making the model a validated reusable-workflow input, with Opus available for opt-in deep reviews.
3. **Reduce the turn budget.** Test lowering `--max-turns` from 20 to 8-10. The prompt already provides a complete, SHA-anchored diff, so most reviews should not need 20 repository-inspection turns.
4. **Bound slow attempts.** Apply a shorter model/job time budget (for example, 10-15 minutes) and fail with a clear summary or fall back to a faster model instead of occupying the runner for up to 45 minutes.
5. **Retry selectively.** Retry transient Bedrock/API failures with bounded backoff, but avoid repeating a full expensive review for deterministic failures such as invalid structured output. A focused formatting-repair attempt may be sufficient for schema-only failures.
6. **Use a size-aware review policy.** Route small and medium PRs through the fast path. Reserve the larger model/turn budget for large diffs, high-risk paths, or an explicit maintainer label.
7. **Trim low-value context.** Keep the complete semantic diff, but avoid spending turns on generated files, lockfiles, or unrelated base-tree exploration where repository-specific policy permits it.

## Proposed first experiment

1. Add latency/input-size telemetry.
2. Run paired reviews on a representative sample using:
- Opus with 20 turns (current baseline)
- Sonnet with 8-10 turns
3. Compare duration, actionable findings, false positives, and failures.
4. If review quality is comparable, make the faster configuration the default and retain an opt-in deep-review mode.
5. Add a bounded timeout and selective retry behavior regardless of the selected model.

## Acceptance criteria

- Claude review latency and input-size metrics are visible in the Actions job summary.
- A representative sample is used to compare latency and review quality before changing the default.
- The selected configuration materially improves median and p90 latency (target: at least 40%) without a material loss of actionable findings.
- A single Claude attempt has a documented upper time bound, and retries cannot silently double that bound.
- The README documents the default review mode and how maintainers request a deeper review, if that option is added.

Contributor guide

Open the contributing guide

Research direction

No specific files or tests are named; start by locating the current Claude job and its retry, timeout, model, and turn-budget configuration in the GitHub Actions workflow. Add the requested job-summary telemetry, then run paired reviews on a representative PR sample and compare latency, review quality, and failure behavior against the acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, github-actions
Domain
ai, ci-cd, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.