Data distribution queue should also keep queues for destinations
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
The data distribution queue currently only keeps queues for the source servers of a relocation. This can lead to data distribution issuing a lot of lot of concurrent relocations that all go to the same destination.
In this scenario, those relocations will be queued on the storage server before getting fetched. This may take a really long time, and during this interval, mutations for this range are being written to both the source and the destination. It is also possible that we could have made a better decision on where to relocation the shard if we waited to start the relocation.
Contributor guide
Research direction
Start by locating the data distribution queue and the relocation scheduling path described in the issue. Trace how source-server queues are maintained and how destination-side queuing affects concurrent relocations. Done means relocations targeting the same destination are appropriately controlled and the scenario is covered by relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100