Azure / Azure/bicep

bicep test returns exit code 0 when compilation fails

Closed Beginner friendly
#20,304 0 comments 0 reactions 0 assignees View on GitHub
Needs: Triage :mag:
Dominant language
Bicep
Stars
3.6k
Forks
830
Avg merge
1d 21m
Merged PRs (30d)
79

Description

**Bicep version**

Built from the main branch at commit 6cff656e29547654af36e260634cf2de9ce63820.

**Describe the bug**

`bicep test` returns exit code `0` when the input file has compilation errors. It also reports that all evaluations passed. This can cause a CI job to pass even though the test file could not be compiled. I expected the command to return a non-zero exit code and not report a successful test run.

**To Reproduce**

Create a `bicepconfig.json` file:

```json
{
"experimentalFeaturesEnabled": {
"testFramework": true,
"asserts": true
}
}
```

Create a `main.bicep` file:

```bicep
test foo 'missing.bicep' = {}
```

Run:

```console
bicep test main.bicep
echo $?
```

The command reports `BCP091` followed by:

```text
All 0 evaluations passed!
```

The exit code is `0`.

**Additional context**

`TestCommand.RunAsync` logs the compilation diagnostics but only uses `testResults.Success` to determine the exit code. When the referenced test file cannot be loaded, no test result is added. An empty result set is then considered successful.

I can submit a fix and regression test for this.

Contributor guide

Open the contributing guide

Research direction

Start with TestCommand.RunAsync and trace how compilation diagnostics and testResults.Success determine the command's exit status. Reproduce the missing.bicep case from the issue, then add a regression test showing compilation failure returns non-zero and does not report a successful run.

Written by the indexing model from the issue text.

Assessment

Domain
cli, testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.