microsoft / microsoft/FluidFramework

ConsensusOrderedCollection duplicates jobs during snapshot catch-up reconciliation

Open
#28,040 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
TypeScript
Stars
4.9k
Forks
586
Avg merge
1d 15h
Merged PRs (30d)
146

Description

Describe the bug

ConsensusOrderedCollection can retain duplicate/inconsistent job state across snapshot/catch-up when the owning client is absent from quorum. The deterministic local-server stress reproduction is seed 0 on the operation distribution in PR #27880.

It fails assertEqualConsensusOrderedCollections at packages/dds/ordered-collection/src/test/fuzzUtils.ts:159 with Data contents should be equal. For datastore-0/channel-8, client 5 contains ["WWvBB", "WWvBB"] while client 0 contains ["WWvBB"].

A temporary implementation that reconciles job owners absent from quorum both on connect and after message processing makes seed 0 pass, confirming the snapshot/catch-up job-tracking reconciliation defect described in PR #27579.

Seed 180 is a distinct remove-member/requeue ordering defect tracked in a separate issue. Historical seed 54 was also attributed to ConsensusOrderedCollection consistency, but has not been proven to share this root cause.

To Reproduce

Steps to reproduce the behavior:

  1. On PR #27880's local-server stress operation distribution, run only seed 0.
  2. Client 3 disconnects.
  3. Client 2 creates datastore-0/channel-8, adds "WWvBB", and acquires it.
  4. Client 5 loads after reconnect/staging operations.
  5. Synchronize at operation 177.
  6. Observe duplicate contents on client 5 and divergent collection state across clients.
Expected behavior

Snapshot/catch-up and quorum changes should reconcile job ownership exactly once. Every client should converge on identical ConsensusOrderedCollection contents without duplicate requeued items.

The fix should include a minimized DDS-level regression test covering the absent-owner quorum transition and a client loading from snapshot/catch-up.

Logs
assertEqualConsensusOrderedCollections
packages/dds/ordered-collection/src/test/fuzzUtils.ts:159
AssertionError: Data contents should be equal

client-5: ["WWvBB", "WWvBB"]
client-0: ["WWvBB"]

Related context:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the deterministic seed-0 reproduction from PR #27880 and inspect packages/dds/ordered-collection/src/test/fuzzUtils.ts:159, where the divergent contents are asserted. Trace snapshot/catch-up and message-processing job ownership reconciliation, then add the requested minimized DDS-level regression test covering an absent owner during a quorum transition and snapshot/catch-up loading; done means all clients converge without duplicate items.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.