Azure / Azure/azure-dev

[Spike] azd validate experience - next steps

Open
#7,817 2 comments 0 reactions 2 assignees Claimed by @vhvb1989 View on GitHub
area/core-cli enhancement needs-discussion
Dominant language
Go
Stars
569
Forks
364
Avg merge
2d 19h
Merged PRs (30d)
136

Description

The goal is to catch problems before provisioning (saving users a 5-10 minute deploy-and-fail cycle) and give them something actionable.

## Tier 1: Actionable suggestions (always present)
Every validation finding should include a concrete next step. Not _"consider changing your region"_ but `azd env set AZURE_LOCATION westus2` (where `westus2` is a region we've confirmed has available capacity for your model) or a direct portal link. Suggestions should inspect the user's context to make the message actionable rather than shipping static strings. This is the foundation of our validation experience. It works for users without Copilot, it works for agents driving azd in no-prompt mode, and it gives tier 3 below something concrete to execute.

## Tier 2: Targeted fixes (high-value problems only)
For select scenarios where the fix is more involved (the poster child being the quota/region picker) we offer something beyond text suggestions. The key principle: use Copilot to help us *write* deterministic fix code, but ship rules that execute without Copilot at runtime. Not "either defer everything to Copilot or do nothing" - can we shift the AI to compile time and ship something repeatable, durable, and useful for everyone? We're not investing in bespoke UX during prototyping (and avoiding inventing new flows), but this is the direction for high-value problems where tier 1 suggestions aren't enough.

## Tier 3: Agentic fixes (Copilot integration)
For users with Copilot setup, we offer to let the agent attempt the fix. This should be mostly free to implement given we've already built the error context and produced a suggestion. The agent just executes what tier 1 already describes. We should prsent all the validation warnings, suggestions (from tier 1), and ask whether the user wants to let the agent fix it. If so, we should generate a plan, run that by the user for approval, and then execute it.

Validation rules and their suggestions should be testable in isolation. Consider structured evals (#7816) to prevent suggestion quality from regressing as rules are added or APIs change.

## UX: inline suggestions, no warning-by-warning prompts

There was an earlier design that walked users through fixing each warning interactivel but it was flagged this as too many questions for a user. The decision: suggestions appear inline with the warning. No interactive prompts per-warning. If there are 20 and it scrolls past, that's fine - you can scroll back. This keeps azd from blocking on these things (important for no-prompt mode and agent-driven flows) while still giving users all of the detail.

## Validation vs. error handling

Validation is the inverse of what we've done for errors. We're just trying to do it up front now. Longer term we should think about this as one system - "early errors vs live errors" - with shared UX and potentially shared fix logic.

Victor flagged a real problem: at validation we have full infrastructure context (resource list, parameters, the built bicep), but at error time we only have the call stack. If we want shared fix logic between "early errors" and "live errors" (especially if we want the agentic error flow to be as good as the preflight flow) we need to persist that validation context somewhere the error handler can reach it later. This is a design problem worth tracking separately, but we should start thinking about these as one system instead of two.

## Action Items

1. Non-agentic POC: quota validation with actionable inline suggestions to evaluate how far we can get without Copilot. It's fine if this includes lots of heuristics - we can use copilot at compile time.

2. Agentic POC: Take the same scenario and use our agentic features to suggest and optionally implement the fix. We can compare the effectiveness and user experience against the Non-agentic version.

We also need to extend validation beyond deployments so we can expose custom validation to extensions, but that's a non-goal here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.