Azure / Azure/azure-rest-api-specs

ARM API Reviewer agent: pin automated review/eval model to a single, deliberately-chosen configuration

Open
#45,944 0 comments 0 reactions 0 assignees View on GitHub
ARM API Reviewer agent
Dominant language
TypeSpec
Stars
3.1k
Forks
5.9k
Avg merge
3d 2h
Merged PRs (30d)
424

Description

## Problem

Before this hardening work, the unattended ARM API Reviewer GitHub Actions workflow and its evals did not consistently pin a single model/reasoning-effort configuration. Model selection drifted between whatever the gh-aw compiler defaulted to and ad hoc overrides, which meant:

- Two runs of the same PR could receive materially different review quality or behavior depending only on which model happened to be resolved that day.
- Local eval runs (Vally) used a different default than the production workflow, so a "passing" eval did not reliably predict production behavior.
- There was no documented fallback policy for what happens when the pinned model or its pricing is unavailable to the compiled workflow runtime.

## Evidence

- Testing `claude-opus-5?effort=high` and `opusplan` in the compiled fork workflow: both resolve at compile time but **fail at runtime** because the pinned firewall/runtime image's pricing catalog has no AI-credit entry for them.
- `gpt-5.6-sol?effort=high` was verified to run end-to-end successfully in a real fork `/arm-review` run.
- Local Vally could not create sessions for `gpt-5.6-sol?effort=high` at all, requiring `claude-sonnet-4.6` as a documented local-only override — an undocumented gap prior to this work.
- `gh-aw` (compiler v0.86.2) only supports `effort` values of `low`/`medium`/`high` (no `xhigh`), and has no context-tier setting — constraints that were not previously documented anywhere in the repo.

## Proposed Solution

1. Pin the automated workflow's `engine.model` (or top-level `model`, since `engine.model` is deprecated) to `gpt-5.6-sol?effort=high` explicitly in `.github/workflows/arm-api-review.md`, recompiling the lock file so the pin is baked into the generated workflow.
2. Document in `documentation/api-reviewer-agent.md` (or an engineering note) that:
- a pricing/runtime failure for a pinned model does **not** automatically fall back to `auto` (gh-aw's catalog alias fallback does not inspect runtime pricing), so any future model change must be validated against a real fork run before merging;
- local Vally evals must use the documented `claude-sonnet-4.6` override when the production model can't create local sessions, and that eval results under the override are not proof the production model behaves identically.
3. Keep interactive (VS Code / GitHub Copilot app) model selection user-controlled — do not pin the interactive agent, since that would remove a capability users rely on and is out of scope for the automated-consistency goal.
4. Add a regression test asserting the workflow source pins the expected model string, so an accidental revert or drift is caught in CI.

## Labels

`ARM API Reviewer agent`

Contributor guide

Open the contributing guide

Research direction

Start with .github/workflows/arm-api-review.md and documentation/api-reviewer-agent.md, then inspect the generated lock file and existing CI tests. Verify the pinned model and reasoning effort, document the local Vally override and runtime-pricing limitation, and ensure a regression test catches configuration drift without changing interactive model selection.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions
Domain
ci-cd, documentation, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.