Azure / Azure/azure-rest-api-specs
ARM API Reviewer agent: pin automated review/eval model to a single, deliberately-chosen configuration
- Dominant language
- TypeSpec
- Stars
- 3.1k
- Forks
- 5.9k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 424
Description
## Problem
Before this hardening work, the unattended ARM API Reviewer GitHub Actions workflow and its evals did not consistently pin a single model/reasoning-effort configuration. Model selection drifted between whatever the gh-aw compiler defaulted to and ad hoc overrides, which meant:
- Two runs of the same PR could receive materially different review quality or behavior depending only on which model happened to be resolved that day.
- Local eval runs (Vally) used a different default than the production workflow, so a "passing" eval did not reliably predict production behavior.
- There was no documented fallback policy for what happens when the pinned model or its pricing is unavailable to the compiled workflow runtime.
## Evidence
- Testing `claude-opus-5?effort=high` and `opusplan` in the compiled fork workflow: both resolve at compile time but **fail at runtime** because the pinned firewall/runtime image's pricing catalog has no AI-credit entry for them.
- `gpt-5.6-sol?effort=high` was verified to run end-to-end successfully in a real fork `/arm-review` run.
- Local Vally could not create sessions for `gpt-5.6-sol?effort=high` at all, requiring `claude-sonnet-4.6` as a documented local-only override — an undocumented gap prior to this work.
- `gh-aw` (compiler v0.86.2) only supports `effort` values of `low`/`medium`/`high` (no `xhigh`), and has no context-tier setting — constraints that were not previously documented anywhere in the repo.
## Proposed Solution
1. Pin the automated workflow's `engine.model` (or top-level `model`, since `engine.model` is deprecated) to `gpt-5.6-sol?effort=high` explicitly in `.github/workflows/arm-api-review.md`, recompiling the lock file so the pin is baked into the generated workflow.
2. Document in `documentation/api-reviewer-agent.md` (or an engineering note) that:
- a pricing/runtime failure for a pinned model does **not** automatically fall back to `auto` (gh-aw's catalog alias fallback does not inspect runtime pricing), so any future model change must be validated against a real fork run before merging;
- local Vally evals must use the documented `claude-sonnet-4.6` override when the production model can't create local sessions, and that eval results under the override are not proof the production model behaves identically.
3. Keep interactive (VS Code / GitHub Copilot app) model selection user-controlled — do not pin the interactive agent, since that would remove a capability users rely on and is out of scope for the automated-consistency goal.
4. Add a regression test asserting the workflow source pins the expected model string, so an accidental revert or drift is caught in CI.
## Labels
`ARM API Reviewer agent`
Contributor guide
Research direction
Start with .github/workflows/arm-api-review.md and documentation/api-reviewer-agent.md, then inspect the generated lock file and existing CI tests. Verify the pinned model and reasoning effort, document the local Vally override and runtime-pricing limitation, and ensure a regression test catches configuration drift without changing interactive model selection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions
- Domain
- ci-cd, documentation, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100