Azure / Azure/bicep

Module registry test is flaky in CI

Open
#11,594 2 comments 0 reactions 0 assignees View on GitHub
tech debt
Dominant language
Bicep
Stars
3.6k
Forks
830
Avg merge
1d 2h
Merged PRs (30d)
79

Description

**Bicep version**
`main`

**Describe the bug**
The `ModuleRestoreContentionShouldProduceConsistentState` test has been failing intermittently in CI with the message:

```
Expected dispatcher.GetModuleRestoreStatus(moduleReference, out _) to be ArtifactRestoreStatus.Succeeded {value: 2}, but found ArtifactRestoreStatus.Failed {value: 1}. at FluentAssertions.Execution.LateBoundTestFramework.Throw(String message)
at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
at FluentAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
at FluentAssertions.Primitives.EnumAssertions`2.Be(TEnum expected, String because, Object[] becauseArgs)
at Bicep.Core.IntegrationTests.RegistryTests.ModuleRestoreContentionShouldProduceConsistentState() in /Users/runner/work/bicep/bicep/src/Bicep.Core.IntegrationTests/RegistryTests.cs:line 205
```

An example of a failed run can be found at https://github.com/Azure/bicep/pull/11476/checks?check_run_id=16084350809

**To Reproduce**
Since the test failure is not deterministic, I don't think there will be a stable repro.

**Additional context**
Each suite of CI checks will run this test four times (once in each environment), and a noticeable percentage of the time, the test will fail in one environment. Rerunning CI will generally clear the error, and there are many runs in which the test succeeded in all environments, so I would guess that the test has a likelihood of failure in the low single digits.

I don't believe I've seen a failure in the `win-x64` environment, so this may require investigation on Linux or OS X.

Contributor guide

Open the contributing guide

Research direction

Start with src/Bicep.Core.IntegrationTests/RegistryTests.cs at line 205 and the ModuleRestoreContentionShouldProduceConsistentState test. Compare the failed run linked in the issue with reruns across the Linux, OS X, and win-x64 environments. Done means the test consistently reports Succeeded without intermittent CI failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.