Module registry test is flaky in CI
- Dominant language
- Bicep
- Stars
- 3.6k
- Forks
- 830
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 79
Description
**Bicep version**
`main`
**Describe the bug**
The `ModuleRestoreContentionShouldProduceConsistentState` test has been failing intermittently in CI with the message:
```
Expected dispatcher.GetModuleRestoreStatus(moduleReference, out _) to be ArtifactRestoreStatus.Succeeded {value: 2}, but found ArtifactRestoreStatus.Failed {value: 1}. at FluentAssertions.Execution.LateBoundTestFramework.Throw(String message)
at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
at FluentAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
at FluentAssertions.Primitives.EnumAssertions`2.Be(TEnum expected, String because, Object[] becauseArgs)
at Bicep.Core.IntegrationTests.RegistryTests.ModuleRestoreContentionShouldProduceConsistentState() in /Users/runner/work/bicep/bicep/src/Bicep.Core.IntegrationTests/RegistryTests.cs:line 205
```
An example of a failed run can be found at https://github.com/Azure/bicep/pull/11476/checks?check_run_id=16084350809
**To Reproduce**
Since the test failure is not deterministic, I don't think there will be a stable repro.
**Additional context**
Each suite of CI checks will run this test four times (once in each environment), and a noticeable percentage of the time, the test will fail in one environment. Rerunning CI will generally clear the error, and there are many runs in which the test succeeded in all environments, so I would guess that the test has a likelihood of failure in the low single digits.
I don't believe I've seen a failure in the `win-x64` environment, so this may require investigation on Linux or OS X.
Contributor guide
Research direction
Start with src/Bicep.Core.IntegrationTests/RegistryTests.cs at line 205 and the ModuleRestoreContentionShouldProduceConsistentState test. Compare the failed run linked in the issue with reruns across the Linux, OS X, and win-x64 environments. Done means the test consistently reports Succeeded without intermittent CI failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100