githubnext / githubnext/ado-aw
[aw]: Allow engine.model to be driven by an ADO pipeline variable / variable group
- Dominant language
- Rust
- Stars
- 23
- Forks
- 8
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 22
Description
### Submission requirements
- [x] I generated this issue with an agent that used `.github/agents/ado-aw.agent.md`.
- [x] I reviewed the generated issue and confirm it is being filed directly in `githubnext/ado-aw`.
### Problem summary
## Description
### Context
We run three compiled copilot-engine agents (`implementer`, `pr-reviewer`, and `pr-comment-fixer`) using the Azure AI Foundry BYOM provider and the standalone target. Their model deployment is an operational fleet-wide setting that belongs naturally in a shared Azure DevOps variable group.
### Problem
In ado-aw 0.50.0, `engine.model` must be a compile-time literal. It cannot be driven by an Azure DevOps pipeline variable or variable group and therefore cannot be switched at queue time without editing each `*.agent.md`, recompiling, and committing each generated `*.agent.lock.yml`.
Source-verified behavior:
- `engine.model` is validated against a character allow-list (`[A-Za-z0-9._:-]`) in `src/engine.rs`, which rejects ADO macro syntax such as `$(VAR)` and `${{ }}`.
- The `COPILOT_MODEL` environment variable and `engine.args` back doors are also blocked as substitution paths.
- Only `engine.provider.base-url` currently accepts a `$(VAR)` macro, and its host must then be added to `network.allowed` manually.
### Real-world impact
Rolling our fleet from `gpt-5.5` to `gpt-5.6-terra` required editing every `.agent.md`, recompiling, and committing every `.agent.lock.yml`. We would rather change one entry in a shared ADO variable group and re-run the pipelines. This would also make A/B testing, rollback, and cost-based retargeting operational changes rather than source-code changes.
### Requested feature
Allow `engine.model`—and ideally the whole `engine` block—to be sourced from an Azure DevOps pipeline variable or variable group, with resolution at pipeline runtime.
Preferred resolutions, in order:
1. Permit ADO macro syntax (`$(VAR)` / variable-group reference) in `engine.model`, resolved at pipeline runtime rather than compile time. Validate literal values only and pass macros through untouched.
2. Add a dedicated explicitly-variable field, for example `engine.model-variable: MY_MODEL_VAR`, which compiles to `--model $(MY_MODEL_VAR)`.
3. Re-enable `COPILOT_MODEL`, or provide an equivalent documented environment override, for the compiled default.
Example desired configuration:
```yaml
engine:
id: copilot
model: $(COPILOT_MODEL_DEPLOYMENT)
```
### Benefits
- One shared variable-group change can retarget all agents.
- Queue-time model selection needs no source edit, recompilation, or lock-file churn.
- Model rollout and rollback become faster and less error-prone.
- The compiled pipeline remains integrity-checkable and reproducible.
- The approach aligns model configuration with the existing runtime-variable support for `engine.provider.base-url`.
### Related issues
- #1261 added related Azure Copilot Foundry BYOM configuration support through `engine.env`, but does not make the selected model runtime-configurable.
- #2030 is a manually-authored equivalent of this request. It was closed as a duplicate in anticipation of this agent-filed issue and can remain closed in favor of this issue.
## Labels
- feature
- copilot
- azure-integration
- engine-configuration
### Reproduction details
**Steps to reproduce:**
1. Create an ado-aw 0.50.0 standalone agent using the copilot engine and Azure AI Foundry BYOM provider.
2. Set `engine.model` to an ADO variable-group macro such as `$(COPILOT_MODEL_DEPLOYMENT)`.
3. Run `ado-aw compile`.
**Expected behavior:**
- Compilation succeeds.
- The generated pipeline passes the macro through to the model argument.
- Azure Pipelines resolves the model from the linked variable group at runtime.
- Changing the variable-group value and re-queuing switches models without source or lock-file changes.
**Actual behavior:**
- Compilation rejects the model because `$`, `(`, and `)` are outside the `engine.model` allow-list.
- `COPILOT_MODEL` and `engine.args` cannot provide a supported override.
- Each model change requires editing, recompiling, and committing every agent pipeline.
**Environment:**
- ado-aw: 0.50.0
- Engine: copilot
- Provider: Azure AI Foundry BYOM
- Target: standalone
- Pipeline configuration: shared Azure DevOps variable group
**Minimal reproduction:**
```yaml
---
name: Runtime model test
engine:
id: copilot
model: $(COPILOT_MODEL_DEPLOYMENT)
---
Run with the model deployment selected by the shared variable group.
```
### Proposed next step
1. Decide whether to support direct ADO macros in `engine.model` or introduce an explicit `model-variable` field.
2. Preserve strict validation for literal model identifiers while safely passing recognized runtime references through unchanged.
3. Compile the selected form to the copilot CLI `--model` argument at pipeline runtime.
4. Add compiler tests for literal models, valid macro references, malformed expressions, and variable-group-backed execution.
5. Document queue-time model switching, variable-group usage, and interaction with provider/network configuration.
6. Consider extending the same explicit runtime-variable mechanism to other safe fields in the `engine` block.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in src/engine.rs and reproduce the failure with the minimal configuration using ado-aw compile. Compare the proposed runtime-variable approaches, then add compiler tests for literal models, valid and malformed macro references, and variable-group-backed execution; document the selected behavior and provider/network interaction when done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, rust
- Domain
- build-system, ci-cd, devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100