Azure / Azure/azure-dev

Support extensions hosting their own functional tests (public test-support module + source-scoped replace)

Open
#8,795 0 comments 0 reactions 0 assignees View on GitHub
area/ext-framework engsys enhancement test automation
Dominant language
Go
Stars
569
Forks
364
Avg merge
2d 19h
Merged PRs (30d)
136

Description

## Summary

Enable azd **extensions to host their own functional tests inside their own extension folder** (e.g. `cli/azd/extensions//test/...`) by exposing a **public, internal-free test-support package** in azd core that extensions can consume **from source** for build/test only.

This is a follow-up direction coming out of #8760 / PR #8754, which added PR-gate functional tests for the `azure.ai.agents` extension. Those tests currently live in `cli/azd/test/functional/ai_agents/` (the azd core module) rather than in the extension folder, and the investigation below explains why and what core changes would unblock co-location.

## Why extension tests can't live in the extension folder today

Each extension is a **separate Go module** (e.g. `module azureaiagent`) that consumes azd core as an external dependency pinned to a released version (e.g. `github.com/azure/azure-dev/cli/azd v1.24.3`), with **no `replace`**.

The functional-test helpers the tests rely on are `cli/azd/test/azdcli` and `cli/azd/test/recording`. Two problems block importing them from an extension module:

### 1. Internal-package barrier (small, easily fixable)

Because an extension module's import paths are `azureaiagent/...` — **outside** the `github.com/azure/azure-dev/cli/azd/` tree — Go's internal-package rule forbids importing any `cli/azd/internal/...` package. Verified empirically:

```
use of internal package github.com/azure/azure-dev/cli/azd/internal/runcontext/agentdetect not allowed
```

The **entire** internal closure of `test/azdcli` + `test/recording` is just two packages:
- `cli/azd/internal/runcontext/agentdetect`
- `cli/azd/internal` (root) — pulled in **only transitively through** `agentdetect`

…and both trace back to **a single line** in `test/azdcli/cli.go`:

```go
cmd.Env = append(cmd.Env, agentdetect.DisableAgentDetectEnvVar+"=1")
```

where `DisableAgentDetectEnvVar = "AZD_DISABLE_AGENT_DETECT"`. Relocating that one constant to a public package removes **both** internal packages from the closure. No need to make the broad root `internal` package public.

### 2. Version-coupling barrier (the real one)

Even with the internal barrier removed, an extension consuming `test/azdcli`/`test/recording` from a **pinned release** of `cli/azd` would always lag the helper code by a release cycle. Test-harness changes (e.g. the `cli.go` token-format fix in PR #8754) aren't available until a `cli/azd` release + dependency bump. That's exactly why the tests currently live in core, sharing the module with the helpers.

## Proposed core change

1. **Carve out a public, internal-free test-support module** for extension functional testing — e.g. `cli/azd/test/extframework` with **its own `go.mod`** — wrapping/relocating the reusable bits of `azdcli`, `recording`, `cmdrecord`, `ostest`. Prerequisite refactor: move `AZD_DISABLE_AGENT_DETECT` (and any other internal-only symbols the helpers touch) into a public package so the module is internal-free.

2. **Let extensions pin it from source via a scoped `replace`.** Important nuance: a `replace` in an extension's `go.mod` is **global to the module**. Replacing the whole `cli/azd` dependency (`replace github.com/azure/azure-dev/cli/azd => ../../`) would also redirect the **production** dependency to source — which is the thing currently forbidden to merge (see `cli/azd/extensions//AGENTS.md`), because release binaries must build against the pinned, tagged `cli/azd`.

Making the test-support code a **separate module** lets the extension `replace` **only that module** from source:

```
// extension go.mod — test-only support pinned from source; production cli/azd pin untouched
require github.com/azure/azure-dev/cli/azd/test/extframework v0.0.0
replace github.com/azure/azure-dev/cli/azd/test/extframework => ../../../test/extframework
```

Because the extension's **production code never imports** the test-support module, this `replace` only affects `go test` builds, not the released binary. **This is the key to the reporter's question** — "always consuming from source is fine" holds *only* when the from-source replace is scoped to a test-only module that production never imports; it does **not** hold for a blanket `replace` of the whole `cli/azd` module.

## Reusable interactive-CLI driver (for interactive Tier 1/Tier 2 tests)

Beyond the non-interactive runner, the module should ship a **reusable interactive-CLI driver** so extensions can test interactive flows (wizards, list-pickers) by *defining tests only*, not by hand-rolling a per-extension PTY harness. This replaces the tmux + Python driver prototyped in #8758 with a Go-native one.

It wraps the Go-native expect stack already in azd's dependency graph (`cli/azd/go.sum`):
- `github.com/creack/pty` — real PTY for the spawned `azd` binary
- `github.com/Netflix/go-expect` — `expect`-style send/expect API (the Go equivalent of Python `pexpect`)
- `github.com/hinshun/vt10x` — terminal emulator that renders ANSI/cursor sequences into a 2D screen (the equivalent of `tmux capture-pane -p`)

Capability parity with the tmux/Python driver was assessed: PTY, literal text, special/arrow keys (raw escape codes wrapped as named-key helpers), rendered-screen capture (`vt10x.State.String()`), substring/regex waiting, live streaming, timeouts/watchdog, and sequential-session flow are all covered — and it removes the detached-tmux-server cleanup problem (the PTY + child are owned by the test process). Only open item: a one-time spike confirming vt10x renders the real `azd ai agent init` wizard identically to a real terminal.

Sketch of the extension-facing API (driver lives in the shared module; tests are just send/expect/assert):

```go
sess := extframework.StartInteractive(t, "ai", "agent", "init") // PTY + vt10x + go-expect, auto-cleanup via t.Cleanup
sess.ExpectString("Select a subscription")
sess.Send(extframework.KeyDown, extframework.KeyEnter) // named-key helpers
sess.ExpectString("Enter a name")
sess.SendLine("my-agent")
sess.ExpectString("added to your azd project successfully")
require.Contains(t, sess.Screen(), "Done") // rendered snapshot (== capture-pane)

out, code := extframework.Run(t, "provision", "--no-prompt") // non-interactive phases reuse the plain runner
```

The driver shares binary resolution (`CLI_TEST_AZD_PATH`) and the recording session with the non-interactive runner, so the **same driver serves Tier 1 (recorded/playback) and Tier 2 (live)**. Any Go extension pins it from source via the test-only `replace` and writes interactive tests with no per-extension harness.

## Acceptance criteria

- [ ] A public, internal-free test-support module exists (own `go.mod`), reusing the existing recording/playback proxy infrastructure.
- [ ] `AZD_DISABLE_AGENT_DETECT` (and any other internal symbols the helpers need) relocated to a public package; `test/azdcli`/`recording` no longer import `cli/azd/internal/...`.
- [ ] An extension (pilot: `azure.ai.agents`) can place functional tests under its own folder, importing the test-support module via a source-scoped `replace`, with the production `cli/azd` dependency still pinned to a release.
- [ ] CI builds/runs the extension's in-folder functional tests (record/playback), with no `replace` of the production `cli/azd` module.
- [ ] Docs updated (`cli/azd/AGENTS.md` + extension `AGENTS.md`) to describe the sanctioned source-scoped test `replace` pattern vs. the forbidden production `replace`.

- [ ] The module exposes a reusable **interactive-CLI driver** (`StartInteractive`/send/expect/`Screen()` + named-key helpers) built on `creack/pty` + `Netflix/go-expect` + `hinshun/vt10x`, with auto-cleanup via `t.Cleanup`.
- [ ] Spike: confirm `vt10x` renders the real `azd ai agent init` wizard identically to a real terminal (parity with `tmux capture-pane`).
- [ ] The interactive driver works for both recorded (Tier 1) and live (Tier 2) runs, sharing `CLI_TEST_AZD_PATH` and the recording session with the non-interactive runner.
- [ ] Pilot: migrate #8758's tmux/Python Tier 2 `init` step to the Go interactive driver to prove parity.

## Notes / open questions

- Confirm the released-binary pipeline never picks up the test-only `replace` (it shouldn't, since production code doesn't import the module, but worth a guard/lint check).
- Decide whether to publish/tag the test-support module or keep it source-only (source-only `replace` is sufficient for build/test and avoids a release cadence dependency).
- Consider a lint/CI check that fails if an extension `go.mod` contains a `replace` of `github.com/azure/azure-dev/cli/azd` itself (the forbidden case), while allowing a `replace` scoped to the test-support module.

Related: #8760, #8754

Contributor guide

Open the contributing guide

Research direction

Start by reading cli/azd/test/azdcli, cli/azd/test/recording, cli/azd/go.sum, and the relevant AGENTS.md files, then inspect the azure.ai.agents extension setup. Done means an internal-free test-support module, a source-scoped test replace that leaves the production pin unchanged, CI coverage for in-folder tests, updated docs, and the interactive-driver spike and pilot.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
build-system, cli, developer-experience, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.