bicep test returns exit code 0 when compilation fails
- Dominant language
- Bicep
- Stars
- 3.6k
- Forks
- 830
- Avg merge
- 1d 21m
- Merged PRs (30d)
- 79
Description
**Bicep version**
Built from the main branch at commit 6cff656e29547654af36e260634cf2de9ce63820.
**Describe the bug**
`bicep test` returns exit code `0` when the input file has compilation errors. It also reports that all evaluations passed. This can cause a CI job to pass even though the test file could not be compiled. I expected the command to return a non-zero exit code and not report a successful test run.
**To Reproduce**
Create a `bicepconfig.json` file:
```json
{
"experimentalFeaturesEnabled": {
"testFramework": true,
"asserts": true
}
}
```
Create a `main.bicep` file:
```bicep
test foo 'missing.bicep' = {}
```
Run:
```console
bicep test main.bicep
echo $?
```
The command reports `BCP091` followed by:
```text
All 0 evaluations passed!
```
The exit code is `0`.
**Additional context**
`TestCommand.RunAsync` logs the compilation diagnostics but only uses `testResults.Success` to determine the exit code. When the referenced test file cannot be loaded, no test result is added. An empty result set is then considered successful.
I can submit a fix and regression test for this.
Contributor guide
Research direction
Start with TestCommand.RunAsync and trace how compilation diagnostics and testResults.Success determine the command's exit status. Reproduce the missing.bicep case from the issue, then add a regression test showing compilation failure returns non-zero and does not report a successful run.
Written by the indexing model from the issue text.
Assessment
- Domain
- cli, testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100