apple / apple/foundationdb

DD: Build teams in batches to avoid slow task from DD

Open
#2,102 3 comments 0 reactions 0 assignees View on GitHub
data distribution good first issue
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

The current DD build all required teams in one big atomic block.

When there are a lot of teams to build, say over 6000 teams, DD will take tens of seconds (say 20 seconds) to build all teams without yield. This leads to slow task which makes DD non-responsive to other requests, such as heart beat. Slow tasks can lead to problems: such as cluster controller can mark a live DD as dead, and data balance can be blocked.

The solution to this issue is to break the non-stopping block of `buildTeams()` into smaller asynchronous blocks. When a lot of teams are to build, `buildTeams()` will build some of them (say 100 teams) and yield. Later, `buildTeams()` build another batch of teams.

This issue is related to Issue #2096.

Contributor guide

Open the contributing guide

Research direction

Start by locating buildTeams() in the DD implementation and read how it currently builds all teams in one atomic block. Review Issue #2096 for related context and trace how asynchronous work can yield between batches. Done means large builds yield between bounded batches while preserving team creation and keeping DD responsive to heartbeats and other requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.