microsoft / microsoft/FluidFramework
ConsensusOrderedCollection requeues removed-member items in inconsistent order
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 4.9k
- Forks
- 586
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 146
Description
Describe the bug
ConsensusOrderedCollection can converge on different item ordering across clients after an acquiring client is removed and its job is requeued. This is distinct from the snapshot/catch-up job-tracking reconciliation defect tracked separately.
The deterministic local-server stress reproduction is seed 180 under the operation distribution in PR #27880. It fails assertEqualConsensusOrderedCollections at packages/dds/ordered-collection/src/test/fuzzUtils.ts:159 with Data contents should be equal.
Baseline divergence for datastore-0/channel-4:
- Client 0:
["uxpfF", "9CCyQ", "9CCyQ", "WWvBB", "9CCyQ", "WWvBB", "WWvBB"] - Client 5: missing the leading
"uxpfF"
Applying job-owner/quorum reconciliation fixes the missing value but not convergence: client 5 then contains "uxpfF" after the first two "9CCyQ" entries while client 0 keeps it first. This confirms a separate remove-member/requeue ordering defect.
To Reproduce
Steps to reproduce the behavior:
- On PR #27880's local-server stress operation distribution, run only seed
180. - Client 2 adds
"uxpfF"todatastore-0/channel-4at operation 128. - Clients 2 and 4 later acquire from channel 4.
- Remove client 4 at operation 152, synchronize, then load client 5 at operation 171.
- Perform final synchronization.
- Observe different
ConsensusOrderedCollectioncontents/order across clients.
Expected behavior
When an acquiring client leaves, requeued items must have one deterministic position on every client, including clients loaded after the removal. All clients should converge on identical ordered contents.
The fix should include a minimized DDS-level regression test covering removal, requeue, snapshot/catch-up, and final ordering.
Logs
Failure:
assertEqualConsensusOrderedCollections
packages/dds/ordered-collection/src/test/fuzzUtils.ts:159
AssertionError: Data contents should be equal
Related context:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with packages/dds/ordered-collection/src/test/fuzzUtils.ts:159 and reproduce seed 180 using the operation distribution from PR #27880. Trace ConsensusOrderedCollection behavior when an acquiring client is removed and its items are requeued, including snapshot/catch-up loading. Done means a minimized DDS-level regression test passes with identical ordering across all clients.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100