dotnet / dotnet/orleans

Flaky stream reliability test reports duplicate publisher after silo restart

Closed
#11,033 5 comments 0 reactions 0 assignees View on GitHub
area-streaming area-testing bug Status: In PR test reliability test-issue
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
14h 42m
Merged PRs (30d)
354

Description

## Summary

`UnitTests.Streaming.Reliability.StreamReliabilityTests.SMS_StreamRel_AllSilosRestart_PubSubCounts` failed in the Azure Storage provider net8.0 lane after all silos restarted. The stream retained one consumer but reported two publishers after the next item was sent.

Observed in PR #10797, whose changes add SQS/Aspire integration and are unrelated to the in-memory stream reliability scenario:

- Run: https://github.com/dotnet/orleans/actions/runs/33760554114
- Job: https://github.com/dotnet/orleans/actions/runs/33760554114/job/100665511284

## Failure

```text
After SendItem - PublisherCount for stream MemoryStreamProvider/StreamReliabilityNamespace/9fda12d0039142b8b05885cdbe932656 = 2; expected 1

at UnitTests.StreamingTests.StreamTestUtils.CheckPubSubCount(...)
at UnitTests.Streaming.Reliability.StreamReliabilityTests.Test_AllSilosRestart_PubSubCounts(...)
at UnitTests.Streaming.Reliability.StreamReliabilityTests.SMS_StreamRel_AllSilosRestart_PubSubCounts()
```

Before the restart, publisher and consumer counts were both 1. After restart, the consumer count remained 1 and the first post-restart send registered the pulling-agent stream, attached the existing subscription, delivered the item, then observed publisher count 2.

## Expected outcome

After all silos restart and the producer resumes sending, the pub/sub registry converges to one publisher and one consumer for the stream. The test should either wait for that deterministic convergence boundary or surface diagnostics identifying both publisher registrations and their ownership.

Contributor guide

Open the contributing guide

Research direction

Start with UnitTests.Streaming.Reliability.StreamReliabilityTests.SMS_StreamRel_AllSilosRestart_PubSubCounts and the CheckPubSubCount failure shown in the report. Reproduce or inspect the Azure Storage provider net8.0 run around the silo restart and first post-restart send. Done means the registry deterministically reports one publisher and one consumer, or diagnostics identify both publisher registrations and their ownership.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.