Azure / Azure/azure-dev

Preserve actionable remediation in fail-fast errors for agent workflows

Open
#9,571 0 comments 0 reactions 1 assignee Claimed by @glharper View on GitHub
ai area/error-handling area/ext-framework bug ext-agents ext-projects
Dominant language
Go
Stars
569
Forks
364
Avg merge
2d 19h
Merged PRs (30d)
136

Description

## Summary

Follow-up to #9570. This issue intentionally excludes the prompt-equivalence, destructive-confirmation, manifest-parameter, eval-context, init-overwrite, and ACR-selection gaps already tracked there.

For agent and CI workflows, fail-fast errors should be self-remediating: name the missing input, list every supported flag/environment/config source, and include an executable example. The remaining gaps below cause coding agents to guess, retry, and consume tokens without making progress.

## Findings

| Priority | Area | Finding |
| --- | --- | --- |
| High | Prompt RPC boundary | `input.PromptRequiredError` already models missing inputs and flag/environment/config sources, but generic extension prompt RPCs create only `PromptMessage`. Across gRPC this degrades to `prompt required`; `azure.ai.agents` then wraps the raw error in `FromPrompt`. The rich input metadata and remediation never reach the extension/user. Also, `PromptRequiredError.ToString` generates executable examples only for environment sources, not flag or config sources. See `cli/azd/pkg/input/prompt_required_error.go`, `cli/azd/internal/grpcserver/prompt_service.go:56-60,95-99,207-210`, and `cli/azd/extensions/azure.ai.agents/internal/exterrors/errors.go:253-276`. |
| High | Environment selection | Several extension lifecycle paths return `no current azd environment is selected`, `current azd environment has no name`, or only suggest `azd env new`. They should name all actual sources: `-e/--environment`, `AZD_ENVIRONMENT`, and `azd env select `, with an example such as `azd -e dev ...`. See `azure.ai.agents/internal/cmd/activity_bot_provision.go:27-32`, `azure.ai.projects/internal/cmd/project_service_config.go:45-55`, and `azure.ai.projects/internal/provisioning/foundry_provisioning_provider.go:641-651`. |
| High | Eval generation | The fail-fast required-input error says one of `--gen-instruction`, `--gen-instruction-file`, `--config`, or dataset/evaluator inputs is required, but `eval generate` does not expose a `--config` flag. It also combines multiple alternatives into one plain error without a complete example command. See `azure.ai.agents/internal/cmd/eval_generate.go:82-94,203-207`. |
| Medium | Azure dependency values | Agent deployment errors correctly name `AZURE_SUBSCRIPTION_ID`, `AZURE_LOCATION`, and `FOUNDRY_PROJECT_ENDPOINT`, but remediation is indirect (`run provision`, `run init`, or `--project-id ...`) instead of giving the exact supported command, such as `azd env set AZURE_LOCATION eastus2`. See `azure.ai.agents/internal/project/service_target_agent.go:301-307,528-533,1873-1878,2613-2618`. The Foundry provider subscription/location errors are the stronger pattern to reuse. |
| Medium | Deployed-agent state | `show`, `monitor`, `files`, and `code download` report that agent name/version could not be resolved, then say to deploy or “check environment values” without printing the expected `AGENT__NAME` / `AGENT__VERSION` keys or a service-specific command. `code download` additionally says to pass the agent name positionally even though its positional argument is a service name. See `azure.ai.agents/internal/cmd/show.go:83-95`, `monitor.go:93-99`, `files.go:85-90`, and `code.go:90-96`. |
| Policy | Plain or empty remediation | Multiple user-fixable paths use plain `fmt.Errorf`, so no structured suggestion reaches the azd host. Several `exterrors.Dependency` calls also pass an empty suggestion, including project initialization/not-found paths. User-fixable `Validation`/`Dependency` errors should not be emitted without remediation. |

## Desired error shape

```text
error: azd environment name is required

Provide one of:
Flag: --environment (or -e )
Environment: AZD_ENVIRONMENT
Config: current environment selection

Examples:
azd ai agent show -e dev
$env:AZD_ENVIRONMENT = "dev" # PowerShell
azd env select dev
```

The concrete sources and examples should reflect what the command actually reads. In particular, the azd environment selector variable is `AZD_ENVIRONMENT`, not `AZURE_ENV_NAME`.

## Acceptance criteria

- Preserve missing-input metadata across the host/extension gRPC boundary.
- Render copy-paste examples for flag, environment, and config input sources.
- Add a shared structured missing-input helper for extensions instead of plain `fmt.Errorf`.
- Environment-selection failures name `--environment/-e`, `AZD_ENVIRONMENT`, and `azd env select`.
- Remove the nonexistent `--config` remediation from `eval generate` or implement the flag.
- Missing Azure values include exact `azd env set ` commands where manual setting is supported.
- Missing deployed-agent state identifies the expected environment keys and the exact service-targeted deployment command.
- Add regression tests that assert the rendered error contains the input name, supported sources, and an executable example.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.