githubnext / githubnext/ado-aw

[aw]: Allow engine.model to be driven by an ADO pipeline variable / variable group

Open
#2,035 0 comments 0 reactions 0 assignees View on GitHub
enhancement rust
Dominant language
Rust
Stars
23
Forks
8
Avg merge
4d 9h
Merged PRs (30d)
22

Description

### Submission requirements

- [x] I generated this issue with an agent that used `.github/agents/ado-aw.agent.md`.
- [x] I reviewed the generated issue and confirm it is being filed directly in `githubnext/ado-aw`.

### Problem summary

## Description

### Context

We run three compiled copilot-engine agents (`implementer`, `pr-reviewer`, and `pr-comment-fixer`) using the Azure AI Foundry BYOM provider and the standalone target. Their model deployment is an operational fleet-wide setting that belongs naturally in a shared Azure DevOps variable group.

### Problem

In ado-aw 0.50.0, `engine.model` must be a compile-time literal. It cannot be driven by an Azure DevOps pipeline variable or variable group and therefore cannot be switched at queue time without editing each `*.agent.md`, recompiling, and committing each generated `*.agent.lock.yml`.

Source-verified behavior:

- `engine.model` is validated against a character allow-list (`[A-Za-z0-9._:-]`) in `src/engine.rs`, which rejects ADO macro syntax such as `$(VAR)` and `${{ }}`.
- The `COPILOT_MODEL` environment variable and `engine.args` back doors are also blocked as substitution paths.
- Only `engine.provider.base-url` currently accepts a `$(VAR)` macro, and its host must then be added to `network.allowed` manually.

### Real-world impact

Rolling our fleet from `gpt-5.5` to `gpt-5.6-terra` required editing every `.agent.md`, recompiling, and committing every `.agent.lock.yml`. We would rather change one entry in a shared ADO variable group and re-run the pipelines. This would also make A/B testing, rollback, and cost-based retargeting operational changes rather than source-code changes.

### Requested feature

Allow `engine.model`—and ideally the whole `engine` block—to be sourced from an Azure DevOps pipeline variable or variable group, with resolution at pipeline runtime.

Preferred resolutions, in order:

1. Permit ADO macro syntax (`$(VAR)` / variable-group reference) in `engine.model`, resolved at pipeline runtime rather than compile time. Validate literal values only and pass macros through untouched.
2. Add a dedicated explicitly-variable field, for example `engine.model-variable: MY_MODEL_VAR`, which compiles to `--model $(MY_MODEL_VAR)`.
3. Re-enable `COPILOT_MODEL`, or provide an equivalent documented environment override, for the compiled default.

Example desired configuration:

```yaml
engine:
id: copilot
model: $(COPILOT_MODEL_DEPLOYMENT)
```

### Benefits

- One shared variable-group change can retarget all agents.
- Queue-time model selection needs no source edit, recompilation, or lock-file churn.
- Model rollout and rollback become faster and less error-prone.
- The compiled pipeline remains integrity-checkable and reproducible.
- The approach aligns model configuration with the existing runtime-variable support for `engine.provider.base-url`.

### Related issues

- #1261 added related Azure Copilot Foundry BYOM configuration support through `engine.env`, but does not make the selected model runtime-configurable.
- #2030 is a manually-authored equivalent of this request. It was closed as a duplicate in anticipation of this agent-filed issue and can remain closed in favor of this issue.

## Labels

- feature
- copilot
- azure-integration
- engine-configuration

### Reproduction details

**Steps to reproduce:**

1. Create an ado-aw 0.50.0 standalone agent using the copilot engine and Azure AI Foundry BYOM provider.
2. Set `engine.model` to an ADO variable-group macro such as `$(COPILOT_MODEL_DEPLOYMENT)`.
3. Run `ado-aw compile`.

**Expected behavior:**

- Compilation succeeds.
- The generated pipeline passes the macro through to the model argument.
- Azure Pipelines resolves the model from the linked variable group at runtime.
- Changing the variable-group value and re-queuing switches models without source or lock-file changes.

**Actual behavior:**

- Compilation rejects the model because `$`, `(`, and `)` are outside the `engine.model` allow-list.
- `COPILOT_MODEL` and `engine.args` cannot provide a supported override.
- Each model change requires editing, recompiling, and committing every agent pipeline.

**Environment:**

- ado-aw: 0.50.0
- Engine: copilot
- Provider: Azure AI Foundry BYOM
- Target: standalone
- Pipeline configuration: shared Azure DevOps variable group

**Minimal reproduction:**

```yaml
---
name: Runtime model test
engine:
id: copilot
model: $(COPILOT_MODEL_DEPLOYMENT)
---

Run with the model deployment selected by the shared variable group.
```

### Proposed next step

1. Decide whether to support direct ADO macros in `engine.model` or introduce an explicit `model-variable` field.
2. Preserve strict validation for literal model identifiers while safely passing recognized runtime references through unchanged.
3. Compile the selected form to the copilot CLI `--model` argument at pipeline runtime.
4. Add compiler tests for literal models, valid macro references, malformed expressions, and variable-group-backed execution.
5. Document queue-time model switching, variable-group usage, and interaction with provider/network configuration.
6. Consider extending the same explicit runtime-variable mechanism to other safe fields in the `engine` block.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/engine.rs and reproduce the failure with the minimal configuration using ado-aw compile. Compare the proposed runtime-variable approaches, then add compiler tests for literal models, valid and malformed macro references, and variable-group-backed execution; document the selected behavior and provider/network interaction when done.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, rust
Domain
build-system, ci-cd, devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.