Azure / Azure/azure-rest-api-specs
ARM API Reviewer agent: Centralize ARM API Reviewer model configuration
- Dominant language
- TypeSpec
- Stars
- 3.1k
- Forks
- 5.9k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 424
Description
## Summary
PR #45841 pinned ARM API Reviewer threat detection to `claude-sonnet-4.6`, but that model is unavailable to the `agentic-workflows` integrator. Threat detection has failed on activated ARM API Reviewer runs while the parent workflow can still appear successful because detection continues on error.
Use `gpt-5.6-sol?effort=high` for both the primary ARM review and threat detection. Declare the value once in a configuration-only shared workflow component, then reference it from both runtime settings.
## Problem
The workflow previously declared the primary and threat-detection models independently:
```yaml
model: gpt-5.6-sol?effort=high
safe-outputs:
threat-detection:
engine:
id: copilot
model: claude-sonnet-4.6
```
This creates two defects:
1. `claude-sonnet-4.6` is not available to the `agentic-workflows` integrator, so threat detection receives HTTP 400 and ends as `failure / parse_error`.
2. Independent declarations drift. A future ARM model upgrade would require finding and updating multiple locations, increasing maintenance cost and regression risk.
## Evidence
- Pre-merge [run 33914570848](https://github.com/Azure/azure-rest-api-specs/actions/runs/33914570848): threat detection completed successfully.
- Post-merge [run 33996004395](https://github.com/Azure/azure-rest-api-specs/actions/runs/33996004395): detector failed with `parse_error`.
- Post-merge [run 34117616401](https://github.com/Azure/azure-rest-api-specs/actions/runs/34117616401): detector failed with `parse_error`.
- Post-merge [run 34133319567](https://github.com/Azure/azure-rest-api-specs/actions/runs/34133319567): four detector attempts returned HTTP 400 because the requested model was unavailable, then ended as `failure / parse_error`.
- The failures were recorded in #45320, but their parent workflow runs still concluded successfully and published safe outputs.
- `gpt-5.6-sol?effort=high` has completed real agentic workflow runs successfully.
- gh-aw `v0.86.2` successfully compiles a local shared component that exports one model environment value consumed by both jobs.
## Scope
### In scope
- One canonical model declaration for the ARM API Reviewer workflow.
- The primary ARM review model.
- ARM threat-detection model.
- ARM generated lock file.
- ARM workflow regression tests.
- A parity assertion that ARM Vally agent models equal the canonical value.
- Mirroring the ARM workflow assets to `Azure/azure-rest-api-specs-pr` when the assignee has access.
### Out of scope
- `.github/workflows/data-plane-api-review.md` and its lock file.
- Vally `judge_model` values.
- Local example model overrides in documentation or runner help.
- Models used by unrelated skills, hero scenarios, or SDK generation.
The existing inventory comment on this issue is context only. It does not expand this task to data-plane or unrelated model declarations.
## File-by-file implementation plan
### 1. Add the canonical shared component
Create:
`.github/workflows/shared-github-aw-imports/arm-api-review-model.md`
Use configuration-only frontmatter:
```yaml
---
# No `on:` here. This is a shared component meant to be imported.
description: Canonical model configuration for the ARM API Reviewer workflow
env:
ARM_API_REVIEWER_MODEL: gpt-5.6-sol?effort=high
---
```
Do not add a Markdown body. gh-aw imports component bodies into the agent prompt, and this file contains configuration rather than reviewer instructions.
### 2. Reference the shared value from the ARM workflow
Update:
`.github/workflows/arm-api-review.md`
Add the local import:
```yaml
imports:
- shared-github-aw-imports/arm-api-review-model.md
```
Set the primary model:
```yaml
engine:
id: copilot
model: ${{ env.ARM_API_REVIEWER_MODEL }}
```
Set the threat-detection model:
```yaml
safe-outputs:
threat-detection:
engine:
id: copilot
model: ${{ env.ARM_API_REVIEWER_MODEL }}
```
Keep the `engine.id` values unchanged. Do not use a YAML anchor because anchors cannot cross files. Do not use a repository variable because the requested source of truth must be versioned and reviewable with the code.
### 3. Regenerate the ARM lock file
Regenerate:
`.github/workflows/arm-api-review.lock.yml`
Use the compiler version recorded in the current lock-file header. For the current implementation this is gh-aw `v0.86.2`.
```powershell
gh aw --version
gh aw compile arm-api-review --no-check-update
```
Expected generated state:
- One top-level `ARM_API_REVIEWER_MODEL: gpt-5.6-sol?effort=high` value.
- Compiler metadata records both agent and detection models as `${{ env.ARM_API_REVIEWER_MODEL }}`.
- Exactly two `COPILOT_MODEL: ${{ env.ARM_API_REVIEWER_MODEL }}` runtime references.
- No unrelated gh-aw action, container, telemetry, or compiler-version upgrade churn.
Do not edit the lock file manually.
### 4. Add deterministic regression coverage
Update:
`.github/workflows/test/arm-api-review-workflow.test.js`
Required test changes:
1. Add a `MODEL_CONFIG_FILE` constant for the shared component.
2. Add a helper that reads `ARM_API_REVIEWER_MODEL` from the component frontmatter.
3. Assert the source workflow imports the component.
4. Assert the source workflow contains exactly two `${{ env.ARM_API_REVIEWER_MODEL }}` model references.
5. Assert the source workflow does not duplicate the concrete model value.
6. Assert the compiled lock contains the canonical environment value.
7. Assert the compiled lock contains exactly two runtime `COPILOT_MODEL` references.
8. Assert the compiled workflow does not fall back to `vars.GH_AW_MODEL_AGENT_COPILOT` for the configured value.
9. Read every ARM Vally eval YAML file and assert its agent `model` equals the canonical shared value. Do not compare `judge_model` to the ARM runtime model.
Tests must derive the expected value from the shared component. Do not duplicate `gpt-5.6-sol?effort=high` in test assertions.
### 5. Preserve repository parity
The ARM workflow also exists in `Azure/azure-rest-api-specs-pr`. If the assignee can access that repository, mirror:
- `shared-github-aw-imports/arm-api-review-model.md`
- `arm-api-review.md`
- `arm-api-review.lock.yml`
- Relevant regression tests
If the assignee cannot access the private repository, complete the public repository implementation and report the private mirror as an explicit follow-up. Lack of private access must not cause unrelated public changes.
## Acceptance criteria
- The shared component is the only ARM workflow configuration file containing `gpt-5.6-sol?effort=high`.
- The ARM source workflow contains no concrete model literal.
- The primary review and threat detection both reference `ARM_API_REVIEWER_MODEL`.
- The shared component has no body and adds no model-configuration prose to the reviewer prompt.
- The generated lock has one canonical environment value and two runtime model references.
- No unrelated gh-aw compiler or runtime dependency churn is included.
- ARM Vally agent model declarations remain equal to the canonical value.
- Vally judge models and data-plane workflow models are unchanged.
- Focused and full deterministic validation pass.
- A real ARM workflow run completes threat detection without HTTP 400 or `parse_error`.
- A clean or no-findings workflow run also completes threat detection successfully.
## Validation commands
```powershell
# Compile with the repository-matching gh-aw version
gh aw --version
gh aw compile arm-api-review --no-check-update
# Focused tests
npx vitest run .github/workflows/test/arm-api-review-workflow.test.js
# Complete GitHub workflow validation
npm --prefix .github run check
# Patch hygiene
git diff --check
git diff -- .github/workflows/arm-api-review.md .github/workflows/arm-api-review.lock.yml .github/workflows/shared-github-aw-imports/arm-api-review-model.md .github/workflows/test/arm-api-review-workflow.test.js
```
After deterministic validation, trigger an eligible test PR with `/arm-review` or `workflow_dispatch`. Confirm both runtime jobs use `gpt-5.6-sol?effort=high` and `GH_AW_DETECTION_CONCLUSION=success`.
## Limitations
- gh-aw `v0.86.2` does not allow an imported component to provide the top-level `model` field directly. The component therefore exports a workflow-level environment value that both model fields reference.
- Because the model fields are GitHub Actions expressions, availability and AI credit pricing still require a real workflow run before adopting a future model.
- gh-aw may generate a fallback environment entry for an expression-based model. The canonical environment value is non-empty, so the explicit model remains the expected runtime selection. Verify this in a real run.
- Vally `v0.14.0` cannot import this shared component into `defaults.model`. Eval YAML files retain model declarations, but deterministic tests compare them with the canonical value and fail on drift.
- Changing the shared component does not automatically regenerate the lock file. The regression test and normal workflow validation must detect a stale lock.
- Using the primary review model for threat detection favors consistency and availability over the lower cost of a separate detector model. Any future decision to use different models should be explicit and should replace, rather than silently bypass, this single-source contract.
Contributor guide
Assessment
This issue has not been assessed yet.