Flaky stream reliability test reports duplicate publisher after silo restart
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 14h 42m
- Merged PRs (30d)
- 354
Description
## Summary
`UnitTests.Streaming.Reliability.StreamReliabilityTests.SMS_StreamRel_AllSilosRestart_PubSubCounts` failed in the Azure Storage provider net8.0 lane after all silos restarted. The stream retained one consumer but reported two publishers after the next item was sent.
Observed in PR #10797, whose changes add SQS/Aspire integration and are unrelated to the in-memory stream reliability scenario:
- Run: https://github.com/dotnet/orleans/actions/runs/33760554114
- Job: https://github.com/dotnet/orleans/actions/runs/33760554114/job/100665511284
## Failure
```text
After SendItem - PublisherCount for stream MemoryStreamProvider/StreamReliabilityNamespace/9fda12d0039142b8b05885cdbe932656 = 2; expected 1
at UnitTests.StreamingTests.StreamTestUtils.CheckPubSubCount(...)
at UnitTests.Streaming.Reliability.StreamReliabilityTests.Test_AllSilosRestart_PubSubCounts(...)
at UnitTests.Streaming.Reliability.StreamReliabilityTests.SMS_StreamRel_AllSilosRestart_PubSubCounts()
```
Before the restart, publisher and consumer counts were both 1. After restart, the consumer count remained 1 and the first post-restart send registered the pulling-agent stream, attached the existing subscription, delivered the item, then observed publisher count 2.
## Expected outcome
After all silos restart and the producer resumes sending, the pub/sub registry converges to one publisher and one consumer for the stream. The test should either wait for that deterministic convergence boundary or surface diagnostics identifying both publisher registrations and their ownership.
Contributor guide
Research direction
Start with UnitTests.Streaming.Reliability.StreamReliabilityTests.SMS_StreamRel_AllSilosRestart_PubSubCounts and the CheckPubSubCount failure shown in the report. Reproduce or inspect the Azure Storage provider net8.0 run around the silo restart and first post-restart send. Done means the registry deterministically reports one publisher and one consumer, or diagnostics identify both publisher registrations and their ownership.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100