Azure / Azure/azure-verified-modules-tools
Terraform testing guidance contradicts itself on `command = apply` for `expect_failures` tests
- Dominant language
- PowerShell
- Stars
- 2
- Forks
- 5
- Avg merge
- 12h 51m
- Merged PRs (30d)
- 30
Description
The Terraform testing reference states a hard rule, then gives an example that cannot satisfy it.
The rule, in `src/Avm.Authoring/.../terraform-test.md` (mirrored into consuming repos as the `avm-tf-testing` skill's `references/terraform-test.md`):
> **Critical rule**: Unit tests use `command = apply` (NOT `command = plan`) because mocked providers make apply safe and allow testing resource creation logic.
And under "Testing Validation Rules":
```hcl
run "test_invalid_location_rejected" {
command = apply
variables {
location = ""
}
expect_failures = [var.location]
}
```
That example does not pass. A variable `validation` block fails during planning, so the apply never executes, and Terraform marks the whole run failed even though the expected failure did occur:
```
run "flex_consumption_requires_runtime_name"... fail
│ Warning: Expected failure while planning
│ A custom condition within var.fc1_runtime_name failed during the planning
│ stage and prevented the requested apply operation. While this was an
│ expected failure, the apply operation could not be executed and so the
│ overall test case will be marked as a failure...
Failure! 0 passed, 1 failed, 10 skipped.
```
The documented example was also reproduced **verbatim** in an isolated scratch module — a single `location` variable with the exact `validation` block and the exact `run` block from the reference — and it fails there too, on Terraform v1.15.8. So this is not an interaction with any particular module's configuration.
### Why this is worth fixing rather than living with
It is self-reinforcing. Anyone who writes a validation test, tries `apply`, watches it fail with a diagnostic that never names the real constraint, and reaches for `plan`, produces a file that quietly violates the stated rule with nothing explaining why.
We hit this concretely while adding unit tests to `Azure/terraform-azurerm-avm-res-web-site`: across eight pull requests, five of seven new test files used `command = plan`, and at least two sessions arrived there independently by exactly that route. A reviewer applying the "Critical rule" literally then asked a contributor to change a working suite to one that could not pass.
### Suggested fix
1. Add an explicit carve-out to the critical rule: `expect_failures` runs targeting variable-level `validation` blocks must use `command = plan`, because variable validation fails during planning and an unexecuted apply marks the run failed.
2. Correct the `test_invalid_location_rejected` example to use `command = plan`.
It is worth stating the boundary precisely, since `expect_failures` against a resource `precondition` or `postcondition` may behave differently from one against a variable `validation`.
_Drafted by Claude Opus 5._
Contributor guide
Research direction
Start with src/Avm.Authoring/.../terraform-test.md and its mirrored references/terraform-test.md, then review the `expect_failures` example and reproduce it with Terraform v1.15.8. Done means the guidance distinguishes variable validation from other conditions and the invalid-location example uses the command that passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- terraform
- Domain
- devops, documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100