Azure / Azure/azure-sdk-for-rust
`test_retry_with_is_retryable` is flaky under verbose Windows CI logging
- Dominant language
- Rust
- Stars
- 884
- Forks
- 365
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 112
Description
## Crate Name
`azure_messaging_eventhubs`
## Crate Version
`0.15.0`
## Description
The `common::retry::tests::test_retry_with_is_retryable` unit test can fail in the Windows stable CI job when verbose logging is enabled.
The test configures `max_total_elapsed` as one second and emits an `info!` event inside the timed retry operation. In the observed failure, that logging call was delayed by concurrent verbose test output long enough for the retry deadline to expire before the first retry:
```text
Maximum retries reached or time elapsed, returning error.
err="please retry"
elapsed=1.2792063s
assertion `left == right` failed
left: "please retry"
right: "I told you not to retry"
```
This is a test timing/logging flake rather than a failure in the buffered producer behavior added by #4873. The retry test is pre-existing; 190 other library tests passed in the failing run.
Failing build: https://dev.azure.com/azure-sdk/29ec6040-b234-4e31-b139-33dc4287b756/_build/results?buildId=6743870&view=logs&jobId=572fa993-8325-50d3-d3dd-24620a5cee49
Relevant test: https://github.com/Azure/azure-sdk-for-rust/blob/e0776143dd3f0a2690dfe7b3cb3257e5064bc1c4/sdk/eventhubs/azure_messaging_eventhubs/src/common/retry.rs#L315-L352
## Steps to Reproduce
1. Run the `azure_messaging_eventhubs` tests on Windows with all features and verbose tracing enabled:
```powershell
cargo test --lib --bins --tests --examples --manifest-path sdk/eventhubs/azure_messaging_eventhubs/Cargo.toml --all-features --no-fail-fast
```
2. Run the tests concurrently with enough logging pressure to delay the `info!` event in `test_retry_with_is_retryable` beyond one second.
3. Observe that `recover_with_backoff` returns the first error because `max_total_elapsed` has expired, causing the assertion to fail.
The failure is timing-dependent and may require repeated or CI-loaded runs.
## Expected Behavior
The unit test should deterministically exercise retry categorization and retry exhaustion without depending on logging throughput or wall-clock scheduling.
## Proposed Fix
Remove the `info!` call from the timed operation and avoid using a one-second wall-clock deadline to assert retry-count behavior. The test could instead use a sufficiently large `max_total_elapsed`, while retaining the small retry delays, so `max_retries` deterministically controls the result.
## Acceptance Criteria
- `test_retry_with_is_retryable` does not emit logging from inside the timed operation.
- The test remains deterministic under verbose, concurrent Windows test execution.
- The test still verifies the expected retry count and final error.
- `cargo test -p azure_messaging_eventhubs --all-features` passes.
## Checklist
- [x] Follow the Code of Conduct.
- [x] Checked for an existing issue describing this failure.
- [x] Included the failing CI run and a minimal explanation of the timing-dependent reproduction.
Contributor guide
Research direction
Start in sdk/eventhubs/azure_messaging_eventhubs/src/common/retry.rs at common::retry::tests::test_retry_with_is_retryable. Run the azure_messaging_eventhubs test command from the issue, including all features, and inspect how the timed operation and max_total_elapsed affect retry exhaustion. Done means the test has no logging inside the timed operation, verifies the expected retry count and final error deterministically, and passes under verbose Windows execution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100