apple / apple/foundationdb

Fetch keys parallelism lock should favor fetches which are closer to completion

Open
#2,338 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

When the storage server fetches a large amount of data, it completes the move in 2MB chunks. Every chuck has to take the Fetch keys parallelism lock, which means the storage servers will make slow progress on all ongoing fetches.

A better strategy would be to continue working on a small set of fetches until completion before starting work on another fetch.

In addition, all of the timings in fetchKeys are related to individual 2MB chunk moves (including `FetchKeysTooLong`). These should be changed to track the overall time.

Contributor guide

Open the contributing guide

Research direction

Start by tracing fetchKeys and the Fetch keys parallelism lock, then inspect how 2MB chunk moves and FetchKeysTooLong timings are recorded. The work is done when near-completion fetches receive preference and timing measurements cover the overall fetch rather than individual chunks.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.