apple / apple/foundationdb

Data distribution queue should also keep queues for destinations

Open
#2,559 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

The data distribution queue currently only keeps queues for the source servers of a relocation. This can lead to data distribution issuing a lot of lot of concurrent relocations that all go to the same destination.

In this scenario, those relocations will be queued on the storage server before getting fetched. This may take a really long time, and during this interval, mutations for this range are being written to both the source and the destination. It is also possible that we could have made a better decision on where to relocation the shard if we waited to start the relocation.

Contributor guide

Open the contributing guide

Research direction

Start by locating the data distribution queue and the relocation scheduling path described in the issue. Trace how source-server queues are maintained and how destination-side queuing affects concurrent relocations. Done means relocations targeting the same destination are appropriately controlled and the scenario is covered by relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.