Fetch keys parallelism lock should favor fetches which are closer to completion
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
When the storage server fetches a large amount of data, it completes the move in 2MB chunks. Every chuck has to take the Fetch keys parallelism lock, which means the storage servers will make slow progress on all ongoing fetches.
A better strategy would be to continue working on a small set of fetches until completion before starting work on another fetch.
In addition, all of the timings in fetchKeys are related to individual 2MB chunk moves (including `FetchKeysTooLong`). These should be changed to track the overall time.
Contributor guide
Research direction
Start by tracing fetchKeys and the Fetch keys parallelism lock, then inspect how 2MB chunk moves and FetchKeysTooLong timings are recorded. The work is done when near-completion fetches receive preference and timing measurements cover the overall fetch rather than individual chunks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100