microsoft / microsoft/FluidFramework

ConsensusOrderedCollection requeues removed-member items in inconsistent order

Open
#28,041 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
TypeScript
Stars
4.9k
Forks
586
Avg merge
1d 15h
Merged PRs (30d)
146

Description

Describe the bug

ConsensusOrderedCollection can converge on different item ordering across clients after an acquiring client is removed and its job is requeued. This is distinct from the snapshot/catch-up job-tracking reconciliation defect tracked separately.

The deterministic local-server stress reproduction is seed 180 under the operation distribution in PR #27880. It fails assertEqualConsensusOrderedCollections at packages/dds/ordered-collection/src/test/fuzzUtils.ts:159 with Data contents should be equal.

Baseline divergence for datastore-0/channel-4:

  • Client 0: ["uxpfF", "9CCyQ", "9CCyQ", "WWvBB", "9CCyQ", "WWvBB", "WWvBB"]
  • Client 5: missing the leading "uxpfF"

Applying job-owner/quorum reconciliation fixes the missing value but not convergence: client 5 then contains "uxpfF" after the first two "9CCyQ" entries while client 0 keeps it first. This confirms a separate remove-member/requeue ordering defect.

To Reproduce

Steps to reproduce the behavior:

  1. On PR #27880's local-server stress operation distribution, run only seed 180.
  2. Client 2 adds "uxpfF" to datastore-0/channel-4 at operation 128.
  3. Clients 2 and 4 later acquire from channel 4.
  4. Remove client 4 at operation 152, synchronize, then load client 5 at operation 171.
  5. Perform final synchronization.
  6. Observe different ConsensusOrderedCollection contents/order across clients.
Expected behavior

When an acquiring client leaves, requeued items must have one deterministic position on every client, including clients loaded after the removal. All clients should converge on identical ordered contents.

The fix should include a minimized DDS-level regression test covering removal, requeue, snapshot/catch-up, and final ordering.

Logs

Failure:

assertEqualConsensusOrderedCollections
packages/dds/ordered-collection/src/test/fuzzUtils.ts:159
AssertionError: Data contents should be equal

Related context:

  • PR #27880
  • PR #27579, which documents a separate ConsensusOrderedCollection job-tracking reconciliation defect

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/dds/ordered-collection/src/test/fuzzUtils.ts:159 and reproduce seed 180 using the operation distribution from PR #27880. Trace ConsensusOrderedCollection behavior when an acquiring client is removed and its items are requeued, including snapshot/catch-up loading. Done means a minimized DDS-level regression test passes with identical ordering across all clients.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.