Make recorded functional tests resilient to dynamic CLI prompts
- Dominant language
- Go
- Stars
- 569
- Forks
- 364
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 136
Description
- [ ] Make sure you've installed the latest version using [instructions](https://github.com/Azure/azure-dev?tab=readme-ov-file#installupgrade-azure-developer-cli)
**Output from `azd version`**
N/A - this concerns the repository's functional test harness.
**Describe the bug**
Recorded functional tests commonly pass fixed newline-delimited input through helpers such as `stdinForProvision()`. The actual prompt sequence can vary by account and environment: multi-tenant users may receive an additional tenant prompt, preconfigured subscription or location values may remove prompts, and provision validation may add a warning confirmation when the principal has conditional role-assignment permissions.
When the sequence changes, a response intended for tenant, subscription, or location selection can be consumed by a later confirmation. Provisioning may then be intentionally canceled with exit code 0, allowing the test to continue until it fails in an unrelated deploy step. Record and playback can also observe different prompt sequences because recording variables populate environment values during playback.
**To Reproduce**
1. Use an account with access to multiple tenants or conditional role-assignment permissions.
2. Run a recorded functional test that calls `azd provision` with `RunCommandWithStdIn(ctx, stdinForProvision(), ...)`.
3. Use a template that contains role assignments so local provision validation can produce a warning.
4. Observe that fixed stdin responses can be shifted to the wrong prompt and provisioning can be canceled before the test continues.
**Expected behavior**
The functional test harness should provide a deterministic way to run provisioning in record and playback modes without depending on the number or order of interactive prompts. Tenant, subscription, and location selection plus the policy for provision-validation warnings should be explicit and consistent across local recording and CI playback.
**Environment**
Information on your environment:
* Language name and version: Go 1.26 functional tests
* IDE and version: N/A
**Additional context**
This surfaced while adding a recorded container-based Function App deployment test for #9250. Potential solutions include a shared provisioning helper that explicitly configures Azure scope and warning behavior, or structured prompt handling that does not rely on positional newline input. The solution should avoid mutating shared user configuration from parallel tests.
Contributor guide
Research direction
Start with the recorded functional tests that call `azd provision` through `RunCommandWithStdIn(ctx, stdinForProvision(), ...)`, and trace how prompts behave in recording and playback. Define and verify deterministic tenant, subscription, location, and provision-validation warning behavior without relying on positional stdin or mutating shared user configuration; confirm the test continues past provisioning in both modes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, go
- Domain
- cli, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100