microsoft / microsoft/FluidFramework
ConsensusOrderedCollection duplicates jobs during snapshot catch-up reconciliation
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 4.9k
- Forks
- 586
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 146
Description
Describe the bug
ConsensusOrderedCollection can retain duplicate/inconsistent job state across snapshot/catch-up when the owning client is absent from quorum. The deterministic local-server stress reproduction is seed 0 on the operation distribution in PR #27880.
It fails assertEqualConsensusOrderedCollections at packages/dds/ordered-collection/src/test/fuzzUtils.ts:159 with Data contents should be equal. For datastore-0/channel-8, client 5 contains ["WWvBB", "WWvBB"] while client 0 contains ["WWvBB"].
A temporary implementation that reconciles job owners absent from quorum both on connect and after message processing makes seed 0 pass, confirming the snapshot/catch-up job-tracking reconciliation defect described in PR #27579.
Seed 180 is a distinct remove-member/requeue ordering defect tracked in a separate issue. Historical seed 54 was also attributed to ConsensusOrderedCollection consistency, but has not been proven to share this root cause.
To Reproduce
Steps to reproduce the behavior:
- On PR #27880's local-server stress operation distribution, run only seed
0. - Client 3 disconnects.
- Client 2 creates
datastore-0/channel-8, adds"WWvBB", and acquires it. - Client 5 loads after reconnect/staging operations.
- Synchronize at operation 177.
- Observe duplicate contents on client 5 and divergent collection state across clients.
Expected behavior
Snapshot/catch-up and quorum changes should reconcile job ownership exactly once. Every client should converge on identical ConsensusOrderedCollection contents without duplicate requeued items.
The fix should include a minimized DDS-level regression test covering the absent-owner quorum transition and a client loading from snapshot/catch-up.
Logs
assertEqualConsensusOrderedCollections
packages/dds/ordered-collection/src/test/fuzzUtils.ts:159
AssertionError: Data contents should be equal
client-5: ["WWvBB", "WWvBB"]
client-0: ["WWvBB"]
Related context:
- Stress distribution PR #26377
- Matrix fix PR #27579, which explicitly leaves this ConsensusOrderedCollection defect out of scope
- Detached blob PR #27880, whose generator change exposes the current deterministic seed
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the deterministic seed-0 reproduction from PR #27880 and inspect packages/dds/ordered-collection/src/test/fuzzUtils.ts:159, where the divergent contents are asserted. Trace snapshot/catch-up and message-processing job ownership reconciliation, then add the requested minimized DDS-level regression test covering an absent owner during a quorum transition and snapshot/catch-up loading; done means all clients converge without duplicate items.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100