Aspire Test Bench: TUI utility for one-step CLI test-session configuration
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 196
Description
## Summary
A small TUI utility — `tools/TestBench` — that lets a developer (or a test author, or someone reproducing a customer report) spin up a **fully-configured terminal session** matching a specific Aspire CLI personality, in one step. Built on Hex1b (the same terminal-automation library the CLI E2E tests use). Composes with the identity sidecar (`docs/specs/cli-identity-sidecar.md`) and the local NuGet proxy (#17823) — Test Bench is the user-facing front door that drives both.
## Problem
Reproducing a customer issue or validating a CLI change today requires assembling the same configuration tuple by hand, every time:
- Which CLI binary? (installed stable? installed staging? dev tree via `dotnet run`? local hive? a specific PR build?)
- Which identity? (channel, version, commit — see identity sidecar spec)
- Which NuGet source? (real nuget.org? local proxy with staged packages? darc staging feed?)
- Which working directory? (a customer's repo? a scratch dir? a template-generated project?)
- Which env vars? (proxy override, channel override, version override, telemetry opt-out for the test, etc.)
- Then start a terminal in that environment, type the right commands, and observe behavior.
Every step is error-prone. Forgetting one env var means the repro silently runs under the wrong identity. The 13.4 bug class (#17527, #17596) is partly downstream of exactly this: configurations that nobody could reliably stand up in isolation, so nobody verified them before release.
## Proposal
`tools/TestBench` — a Hex1b TUI that:
1. **Prompts for a test-session configuration** via a form-style UI:
- **Bits source**: dev tree (`dotnet run --project src/Aspire.Cli`), installed CLI at a given path, a downloaded PR build, a local hive, or a `get-aspire-cli.{sh,ps1} --quality …` install on demand.
- **Identity**: channel / version / commit overrides (via the identity sidecar's `ASPIRE_CLI_*` env vars).
- **NuGet source**: real public feeds, a local proxy with a chosen staged-package directory (#17823), or a custom service-index URL.
- **Working directory**: a chosen path, a fresh temp dir, or a temp dir pre-seeded from `aspire new `.
- **Telemetry**: opt-out by default for test sessions to keep the corpus clean.
2. **Spawns a terminal session** in that environment. The user lands in a real shell with everything wired up, ready to type `aspire …` commands and observe behavior interactively. The session is throwaway: closing it tears down the temp dir, the local proxy, any extracted hive, etc.
3. **Optional scripted scenarios** drive the experience automatically. Test Bench ships with a library of named scenarios — `init-stable-aspire`, `add-azure-storage-staging`, `update-prerelease-to-ga`, `repro-17527`, etc. — that are Hex1b scripts authored in the same idiom as the CLI E2E tests. Selecting one drives the terminal with the right keystrokes, asserts expected screen states, and reports pass/fail. Useful for both manual exploratory testing (watch the scenario, learn what it does) and CI-style validation (run the scenario set against a PR build, get a report).
4. **Session capture**: every Test Bench session can be saved as a Hex1b script. "I just reproduced the bug — save this as `repro-17891`." That script lands in `tools/TestBench/scenarios/` and is replayable on demand. Customer-reported repros become regression scenarios in two clicks.
## Why this exists separately from the existing CLI E2E test infrastructure
- **CLI E2E tests** (`tests/Aspire.Cli.EndToEnd.Tests`) are xUnit-driven, automated, and run in CI. They prove things stay working. They are not interactive — a contributor cannot easily pause one, poke around, and resume.
- **Test Bench** is interactive-first. It exists to *generate* repros, to drive exploratory testing, and to let humans validate identity-conditional behavior with their eyes. Saved sessions can be promoted into the E2E test suite once they earn their keep.
The shared substrate is Hex1b, so a Test Bench session and an E2E test exercise the CLI through identical machinery — what you see in Test Bench is what CI will see.
## Composes with
- **Identity sidecar** (`docs/specs/cli-identity-sidecar.md`): Test Bench's "identity" form fields are exactly the `ASPIRE_CLI_CHANNEL` / `ASPIRE_CLI_VERSION` / `ASPIRE_CLI_COMMIT` env vars the sidecar spec defines.
- **Local NuGet proxy** (#17823): Test Bench's "NuGet source" picker starts/stops the proxy and threads the service-index URL through the same `ASPIRE_CLI_NUGET_SERVICE_INDEX` env var, with the package-directory chosen via the form.
- **Hex1b** (`.agents/skills/hex1b`): the TUI itself and the scripted-scenario engine are both Hex1b.
- **Existing CLI E2E tests**: scenarios saved from Test Bench can be lifted into `tests/Aspire.Cli.EndToEnd.Tests/` with no rewrite.
## Validation patterns this unlocks
1. **One-step customer repro.** A customer reports an issue against stable 13.4.0 with a custom NuGet feed. Open Test Bench, pick `bits=dev-tree`, `channel=stable`, `version=13.4.0`, `commit=`, `nuget=proxy-with-customer's-staged-packages`, `workdir=customer's-repo-clone`. Hit start. Land in a terminal with everything wired. No multi-line env-var prefix, no forgotten flag.
2. **Pre-ship shiproom validation.** Before promoting staging → stable, the shiproom owner runs Test Bench's `pre-stable-13.5.0` scenario set: every scenario runs the dev binary under `channel=stable, version=13.5.0` against the local proxy seeded with the GA candidate packages. Pass/fail report in minutes. This is the workflow that would have caught #17527 and #17596 in the cycle they shipped.
3. **Exploratory testing of identity-conditional code.** A contributor adding a new channel-conditional branch opens Test Bench, runs the same scenario under each channel, watches the diff in behavior. If the diff is wrong, they catch it before code review.
4. **Regression authoring.** Customer reports a bug. The triager opens Test Bench, reproduces, saves the session as a Hex1b scenario. The scenario lands in `tools/TestBench/scenarios/` and runs in every subsequent Test Bench session for that area. Repros become regressions automatically.
## Sketch of the UX
```
┌─ Aspire Test Bench ─────────────────────────────────────────┐
│ │
│ Bits source ▾ dev-tree (src/Aspire.Cli) │
│ ▾ installed: /usr/local/bin/aspire │
│ ▾ PR build (download #17823) │
│ ▾ local hive: ~/.aspire/hives/staging │
│ │
│ Identity channel: ▾ stable │
│ version: [13.5.0_____] │
│ commit: [auto from version_] │
│ │
│ NuGet source ▾ real nuget.org │
│ ▾ local proxy + dir: [~/staged-pkgs____] │
│ ▾ custom service index: [_______________] │
│ │
│ Working dir ▾ temp (fresh) │
│ ▾ temp from template: [aspire-starter_] │
│ ▾ existing: [_______________________] │
│ │
│ Telemetry [x] opt-out for this session │
│ │
│ Scenario ▾ (none — interactive shell) │
│ ▾ init-stable-aspire │
│ ▾ add-azure-storage-staging │
│ ▾ repro-17527 │
│ ▾ ... (saved scenarios) │
│ │
│ [Start session] [Save preset] [Quit] │
└─────────────────────────────────────────────────────────────┘
```
## Open questions
1. **Form factor.** Standalone executable (`dotnet tool install -g Aspire.TestBench`) or contributor-only (`dotnet run --project tools/TestBench`)? Lean: contributor-only first, promote to a global tool if it proves useful outside the contributor loop.
2. **Scenario library structure.** Where do saved scenarios live, how are they versioned, who maintains them? Lean: `tools/TestBench/scenarios/*.hex1b` checked into the repo, scenarios named after the issue or scenario they exist for, contributor-maintained the same way E2E tests are.
3. **Cross-platform reach.** Does the TUI need full Windows console parity, or is "works in Windows Terminal / WSL / macOS Terminal / Linux terminals that Hex1b already supports" sufficient? Lean: track Hex1b's platform matrix.
4. **PR build acquisition flow.** Reuse `eng/scripts/get-aspire-cli-pr.{sh,ps1}` directly, or wrap a smaller "fetch + extract to temp" path inline? Reuse is cheaper; wrapping gives Test Bench a cleaner contract over the binary.
5. **Lifecycle of the spawned terminal session.** Detached process, embedded sub-pane inside Test Bench, or new OS terminal window? Lean: spawn a real OS terminal so the user can resize, attach a debugger, and behave normally — Test Bench's job ends once the env is set up.
## Scope hint
This is a contributor/test-author utility, not a customer-facing feature. It lives in `tools/` next to `QuarantineTools`. It does not ship in any CLI nupkg. It depends on a built CLI (so it's a downstream consumer of the resolver + identity-sidecar work, not a prerequisite for it).
## Related
- `docs/specs/cli-identity-sidecar.md` — drives Test Bench's identity form fields.
- #17823 — drives Test Bench's NuGet-source form fields.
- #17580 — broader packaging rethink; Test Bench validates against the bits this work produces.
- #17527, #17596 — the kind of bugs Test Bench's shiproom-validation pattern would catch pre-release.
- `.agents/skills/hex1b` — the substrate.
- `tests/Aspire.Cli.EndToEnd.Tests/` — Test Bench scenarios graduate here once they earn it.
Contributor guide
Assessment
This issue has not been assessed yet.