Azure / Azure/azure-dev

Require --agent for no-prompt eval commands with multiple agent services

Open
#9,797 0 comments 0 reactions 0 assignees View on GitHub
area/evals bug engineering item ext-agents
Dominant language
Go
Stars
569
Forks
364
Avg merge
2d 19h
Merged PRs (30d)
136

Description

## Description

`azd ai agent eval run` and `azd ai agent eval update` currently use agent resolution with `requireAgent=false`. In `--no-prompt` mode, when a project contains multiple `azure.ai.agent` services and `--agent` is omitted, the ambiguity error is discarded and the commands fall back to the workspace root.

This allows a headless command to continue with unintended context instead of failing deterministically.

## Expected behavior

When `--no-prompt` is used with multiple `azure.ai.agent` services, both commands should fail with an actionable error requiring the caller to provide `--agent`.

## Acceptance criteria

- `eval run` and `eval update` reject ambiguous agent selection in no-prompt mode.
- The error tells the caller to provide `--agent`.
- Regression tests cover multiple agent services without `--agent` in no-prompt mode.

## Context

Follow-up from #9720: https://github.com/Azure/azure-dev/pull/9720#discussion_r3893183342

Contributor guide

Open the contributing guide

Research direction

Start at the `azd ai agent eval run` and `azd ai agent eval update` command entry points and trace agent resolution in `--no-prompt` mode. Add regression coverage for multiple `azure.ai.agent` services without `--agent`; done means both commands fail deterministically with an actionable message requiring `--agent`.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go
Domain
ai, cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.