microsoft / microsoft/aspire

Migrate E2E tests to Hex1bTerminalAutomator API and build reusable automation helpers

Open
#15,241 5 comments 0 reactions 1 assignee Claimed by @mitchdenny View on GitHub
area-engineering-systems
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 12h
Merged PRs (30d)
201

Description

## Overview

Migrate all CLI and Deployment E2E tests from the `Hex1bTerminalInputSequenceBuilder` (fluent builder) API to the `Hex1bTerminalAutomator` (async/await imperative) API, and build reusable automation helpers to streamline test development.

The automator API provides better error handling (immediate fail-fast), improved readability (async/await vs fluent chains), easier debugging (stack traces), and a more natural foundation for building higher-level helpers.

### Current State
- **1 test** uses the new `Hex1bTerminalAutomator` API (`WaitCommandTests`)
- **52 tests** use the old `Hex1bTerminalInputSequenceBuilder` API
- **11 tests** don't use Hex1b directly (MCP tests use MCP SDK, auth tests are static)
- **83 total test methods** across 49 test files

---

## Reusable Automation Helpers to Build

These helpers should be built as extension methods on `Hex1bTerminalAutomator` in shared helper files, combining the automator API with custom input sequences where necessary.

### Core Shell Interaction
- [x] **Prompt waiting helpers** — `WaitForSuccessPromptAsync`, `WaitForErrorPromptAsync`, `WaitForAnyPromptAsync`, `WaitForSuccessPromptFailFastAsync` *(already exists in `Hex1bAutomatorTestHelpers.cs`)*
- [ ] **Command execution helper** — `RunCommandAsync(command, counter)` that types a command, presses enter, and waits for prompt — the most common 3-line pattern in every test
- [ ] **Environment variable helper** — `SetEnvironmentVariableAsync(name, value, counter)` for `export VAR=value` pattern

> **Note (PR #15244):** `RunCommandAsync` was not created. All tests inline the `TypeAsync` → `EnterAsync` → `WaitForSuccessPromptAsync` pattern directly. This is the single highest-value helper remaining — it would deduplicate 600+ lines across CLI and deployment tests.

### CLI Installation & Environment
- [x] **Docker environment setup** — `PrepareDockerEnvironmentAsync` *(exists in `CliE2EAutomatorHelpers.cs`)* — prompt counting, umask, env vars
- [ ] **CLI installation (source build)** — `InstallAspireCliFromSourceAsync(counter)` — build from local source
- [x] **CLI installation (PR build)** — `InstallAspireCliFromPullRequestAsync(prNumber, counter)` — download and install from PR artifact
- [x] **CLI installation (GA release)** — `InstallAspireCliReleaseAsync(counter)` — install from aka.ms
- [x] **Bundle installation** — `InstallAspireBundleFromPullRequestAsync(prNumber, counter)` — for TypeScript tests
- [x] **Source CLI environment** — `SourceAspireCliEnvironmentAsync(counter)` — PATH and env var setup
- [x] **Source bundle environment** — `SourceAspireBundleEnvironmentAsync(counter)` — bundle-specific env

> **Note (PR #15244):** Docker install mode detection (`InstallAspireCliInDockerAsync`) handles source build, GA release, and PR build modes within a single method. The deployment helpers have separate methods for PR and release installs. `InstallAspireCliFromSourceAsync` as a standalone helper was not created — it's handled by the Docker install mode enum.

### Project Scaffolding
- [x] **`aspire new` wizard** — `AspireNewAsync(projectName, counter, template, useRedisCache)` *(exists in `Hex1bAutomatorTestHelpers.cs`)* — interactive template selection, project naming, Redis prompt
- [x] **`aspire init` flow** — `AspireInitAsync(counter)` — single-file AppHost creation
- [x] **Agent init prompt handler** — `DeclineAgentInitPromptAsync(counter)` *(exists)* — handles the agent init confirmation
- [ ] **Generic list selection helper** — A robust helper that finds the top of a selection list, traverses items looking for the desired option (using pattern matching), and detects end-of-list. This replaces hardcoded `DownAsync()` sequences that break when list order changes.

> **Note (PR #15244):** `AspireNewAsync` uses a hardcoded `switch` with explicit `DownAsync()` calls per template enum value. The generic list selection helper was not built — template selection still relies on positional navigation.

### Package Management
- [ ] **`aspire add` (non-interactive)** — `AspireAddPackageAsync(packageName, counter)` — simple package add
- [ ] **`aspire add` (interactive with version select)** — `AspireAddPackageInteractiveAsync(packageName, counter)` — handles the version selection prompt that appears in CI
- [ ] **`aspire add` (interactive with integration select)** — handles dual prompts when multiple packages match
- [ ] **`aspire update` flow** — `AspireUpdateAsync(counter, channel)` — handles the multi-prompt update confirmation flow ("Perform updates?", "NuGet.config file?", "Apply changes?")

> **Note (PR #15244):** All package management flows are inlined directly in each test. The patterns are consistent enough to extract — `aspire add` with version selection appears in ~10 deployment tests.

### AppHost Execution
- [ ] **`aspire run`** — `AspireRunAsync(counter, timeout)` — run and wait for dashboard URL
- [ ] **`aspire start`** — `AspireStartAsync(counter, timeout)` — start detached and wait for dashboard URL
- [ ] **`aspire stop`** — `AspireStopAsync(counter)` — stop running AppHost
- [ ] **`aspire wait`** — `AspireWaitAsync(counter, timeout)` — wait for healthy resources

### Resource Inspection
- [ ] **`aspire describe`** — `AspireDescribeAsync(resourceName, counter, format)` — table and JSON output
- [ ] **`aspire ps`** — `AspirePsAsync(counter, format)` — process listing with optional JSON format
- [ ] **`aspire logs`** — `AspireLogsAsync(resourceName, counter)` — log streaming

### Secrets Management
- [ ] **Secret CRUD** — `AspireSecretSetAsync`, `AspireSecretGetAsync`, `AspireSecretListAsync`, `AspireSecretDeleteAsync` — full secret lifecycle

### Certificates
- [ ] **`aspire cert trust`** — `AspireCertTrustAsync(counter)` — certificate trust flow
- [ ] **`aspire cert clean`** — `AspireCertCleanAsync(counter)` — certificate cleanup

### Deployment
- [ ] **`aspire deploy`** — `AspireDeployAsync(counter, timeout, clearCache)` — deploy and wait for `PIPELINE SUCCEEDED` or `PIPELINE FAILED`, fail-fast on failure
- [ ] **`aspire publish`** — `AspirePublishAsync(counter, format)` — Kubernetes manifest generation
- [ ] **Docker Compose deploy** — `DockerComposeDeployAsync(counter)` — Docker Compose specific flow
- [ ] **Endpoint verification with retry** — `VerifyEndpointAsync(url, maxAttempts, delaySeconds)` — curl retry loop pattern used in all deployment tests

### AppHost File Modification
- [ ] **File modification callback** — `ModifyAppHostFileAsync(filePath, findText, replaceText)` — the `ExecuteCallback` pattern for modifying AppHost.cs/apphost.cs between commands

> **Note (PR #15244):** File modifications are inlined directly using `File.ReadAllText`/`File.WriteAllText` between automator calls. This is actually cleaner with the async/await API than the old `ExecuteCallback` pattern since the code flows naturally. A helper might still be useful for the common `builder.Build().Run()` replacement pattern used in ~15 deployment tests.

### Azure Infrastructure (Deployment Tests)
- [ ] **Azure resource verification** — `VerifyAzureResourceExistsAsync(resourceType, resourceGroup)` — generic az CLI verification
- [x] **Resource group management** — `GenerateResourceGroupName(testCase)`, `TriggerCleanupResourceGroup()` — consistent RG lifecycle
- [ ] **Azure provider registration** — `RegisterAzureProviderAsync(providerNamespace, counter)` — `az provider register --wait` pattern

> **Note (PR #15244):** Resource group management was already in `DeploymentE2ETestHelpers` as static methods and wasn't changed — it doesn't need to be on the automator.

### Diagnostics & Reporting
- [x] **Terminal recording** — Ensure all automator-based tests produce asciinema-compatible recordings
- [x] **Deployment reporting** — `ReportDeploymentSuccess/Failure/CleanupStatus` integration with GitHub step summary

> **Note (PR #15244):** Recording and reporting were already working via `CreateTestTerminal()` and `DeploymentReporter` — no changes needed.

---

## Update Skill Documents

- [x] Update `.github/skills/cli-e2e-testing.md` to document the new `Hex1bTerminalAutomator`-based conventions, helper methods, and patterns so that AI agents can write tests using the new API
- [x] Update `.github/skills/deployment-e2e-testing.md` to cover deployment test patterns with the automator API
- [x] Add examples of the recommended test structure using async/await helpers

---

## CLI E2E Test Migration Task List

### Already Migrated ✅
- [x] `WaitCommandTests` — `CreateStartWaitAndStopAspireProject`

### To Migrate

#### Smoke & Template Tests
- [x] `SmokeTests` — `CreateAndRunAspireStarterProject`
- [x] `BundleSmokeTests` — `CreateAndRunAspireStarterProjectWithBundle`
- [x] `JsReactTemplateTests` — `CreateAndRunJsReactProject`
- [x] `PythonReactTemplateTests` — `CreateAndRunPythonReactProject`
- [x] `TypeScriptStarterTemplateTests` — `CreateAndRunTypeScriptStarterProject`
- [x] `EmptyAppHostTemplateTests` — `CreateEmptyAppHostProject`

#### Start/Stop Tests
- [x] `StartStopTests` — `CreateStartAndStopAspireProject`
- [x] `StartStopTests` — `StopWithNoRunningAppHostExitsSuccessfully`
- [x] `StartStopTests` — `AddPackageWhileAppHostRunningDetached`
- [x] `StartStopTests` — `AddPackageInteractiveWhileAppHostRunningDetached`
- [x] `StopNonInteractiveTests` — `StopNonInteractiveSingleAppHost`
- [x] `StopNonInteractiveTests` — `StopAllAppHostsFromAppHostDirectory`
- [x] `StopNonInteractiveTests` — `StopAllAppHostsFromUnrelatedDirectory`
- [x] `StopNonInteractiveTests` — `StopNonInteractiveMultipleAppHostsShowsError`

#### Resource Inspection Tests
- [x] `DescribeCommandTests` — `DescribeCommandShowsRunningResources`
- [x] `DescribeCommandTests` — `DescribeCommandResolvesReplicaNames`
- [x] `PsCommandTests` — `PsCommandListsRunningAppHost`
- [x] `PsCommandTests` — `PsFormatJsonOutputsOnlyJsonToStdout`
- [x] `LogsCommandTests` — `LogsCommandShowsResourceLogs`
- [x] `MultipleAppHostTests` — `DetachFormatJsonProducesValidJson`

#### Secrets Tests
- [x] `SecretDotNetAppHostTests` — `SecretCrudOnDotNetAppHost`
- [x] `SecretTypeScriptAppHostTests` — `SecretCrudOnTypeScriptAppHost`

#### Package Management Tests
- [x] `CentralPackageManagementTests` — `AspireUpdateRemovesAppHostPackageVersionFromDirectoryPackagesProps`
- [x] `CentralPackageManagementTests` — `AspireAddPackageVersionToDirectoryPackagesProps`

#### Certificate Tests
- [x] `CertificatesCommandTests` — `CertificatesTrust_WithUntrustedCert_TrustsCertificate`
- [x] `CertificatesCommandTests` — `CertificatesClean_RemovesCertificates`
- [x] `CertificatesCommandTests` — `CertificatesTrust_WithNoCert_CreatesAndTrustsCertificate`

#### Agent & Doctor Tests
- [x] `AgentCommandTests` — `AgentCommands_AllHelpOutputs_AreCorrect`
- [x] `AgentCommandTests` — `AgentInitCommand_MigratesDeprecatedConfig`
- [x] `AgentCommandTests` — `DoctorCommand_DetectsDeprecatedAgentConfig`
- [x] `AgentCommandTests` — `AgentInitCommand_DefaultSelection_InstallsSkillOnly`
- [x] `PlaywrightCliInstallTests` — `AgentInit_InstallsPlaywrightCli_AndGeneratesSkillFiles`
- [x] `DoctorCommandTests` — `DoctorCommand_WithoutSslCertDir_ShowsPartiallyTrusted`
- [x] `DoctorCommandTests` — `DoctorCommand_WithSslCertDir_ShowsTrusted`

#### Banner Tests
- [x] `BannerTests` — `Banner_DisplayedOnFirstRun`
- [x] `BannerTests` — `Banner_DisplayedWithExplicitFlag`
- [x] `BannerTests` — `Banner_NotDisplayedWithNoLogoFlag`

#### Docker Deployment Tests
- [x] `DockerDeploymentTests` — `CreateAndDeployToDockerCompose`
- [x] `DockerDeploymentTests` — `CreateAndDeployToDockerComposeInteractive`

#### Kubernetes Tests
- [x] `KubernetesPublishTests` — `CreateAndPublishToKubernetes`

#### TypeScript Tests
- [x] `TypeScriptCodegenValidationTests` — `RestoreGeneratesSdkFiles`
- [x] `TypeScriptCodegenValidationTests` — `RunWithMissingAwaitShowsHelpfulError`
- [x] `TypeScriptPolyglotTests` — `CreateTypeScriptAppHostWithViteApp`
- [x] `ProjectReferenceTests` — `TypeScriptAppHostWithProjectReferenceIntegration`

#### Channel Tests
- [x] `StagingChannelTests` — `StagingChannel_ConfigureAndVerifySettings_ThenSwitchChannels`

---

## Deployment E2E Test Migration Task List

#### ACA (Azure Container Apps) Tests
- [x] `AcaStarterDeploymentTests` — `DeployStarterTemplateToAzureContainerApps`
- [x] `AcaCustomRegistryDeploymentTests` — `DeployStarterTemplateWithCustomRegistry`
- [x] `AcaExistingRegistryDeploymentTests` — `DeployStarterTemplateWithExistingRegistry`
- [x] `AcaCompactNamingDeploymentTests` — `DeployWithCompactNamingFixesStorageCollision`
- [x] `AcaCompactNamingUpgradeDeploymentTests` — `UpgradeFromGaToDevDoesNotDuplicateStorageAccounts`
- [x] `AcaDeploymentErrorOutputTests` — `DeployWithInvalidLocation_ErrorOutputIsClean`

#### AKS (Azure Kubernetes) Tests
- [x] `AksStarterDeploymentTests` — `DeployStarterTemplateToAks`
- [x] `AksStarterWithRedisDeploymentTests` — `DeployStarterTemplateWithRedisToAks`

#### App Service Tests
- [x] `AppServiceReactDeploymentTests` — `DeployReactTemplateToAzureAppService`
- [x] `AppServicePythonDeploymentTests` — `DeployPythonFastApiTemplateToAzureAppService`

#### Azure Resource Tests
- [x] `AzureStorageDeploymentTests` — `DeployAzureStorageResource`
- [x] `AzureContainerRegistryDeploymentTests` — `DeployAzureContainerRegistryResource`
- [x] `AzureKeyVaultDeploymentTests` — `DeployAzureKeyVaultResource`
- [x] `AzureServiceBusDeploymentTests` — `DeployAzureServiceBusResource`
- [x] `AzureEventHubsDeploymentTests` — `DeployAzureEventHubsResource`
- [x] `AzureAppConfigDeploymentTests` — `DeployAzureAppConfigResource`
- [x] `AzureLogAnalyticsDeploymentTests` — `DeployAzureLogAnalyticsResource`

#### ACR Purge Tests
- [x] `AcrPurgeTaskDeploymentTests` — `DeployPythonStarterWithPurgeTask`

#### Language-Specific Deployment Tests
- [x] `PythonFastApiDeploymentTests` — `DeployPythonFastApiTemplateToAzureContainerApps`
- [x] `TypeScriptExpressDeploymentTests` — `DeployTypeScriptExpressTemplateToAzureContainerApps`

#### VNet Private Endpoint Tests
- [x] `VnetKeyVaultInfraDeploymentTests` — `DeployVnetKeyVaultInfrastructure`
- [x] `VnetKeyVaultConnectivityDeploymentTests` — `DeployStarterTemplateWithKeyVaultPrivateEndpoint`
- [x] `VnetSqlServerInfraDeploymentTests` — `DeployVnetSqlServerInfrastructure`
- [x] `VnetSqlServerConnectivityDeploymentTests` — `DeployStarterTemplateWithSqlServerPrivateEndpoint`
- [x] `VnetStorageBlobInfraDeploymentTests` — `DeployVnetStorageBlobInfrastructure`
- [x] `VnetStorageBlobConnectivityDeploymentTests` — `DeployStarterTemplateWithStorageBlobPrivateEndpoint`

---

## Notes

- MCP docs tests (`McpDocsE2ETests` — 11 methods) use the MCP SDK client directly and don't need Hex1b migration
- `AuthenticationTests` (4 methods) are static validation tests without terminal automation
- Migration should be done incrementally, building shared helpers as patterns emerge from converting tests
- The generic list selection helper is a high priority — it replaces the fragile hardcoded `DownAsync()` navigation used in template selection

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.