DD: Build teams in batches to avoid slow task from DD
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
The current DD build all required teams in one big atomic block.
When there are a lot of teams to build, say over 6000 teams, DD will take tens of seconds (say 20 seconds) to build all teams without yield. This leads to slow task which makes DD non-responsive to other requests, such as heart beat. Slow tasks can lead to problems: such as cluster controller can mark a live DD as dead, and data balance can be blocked.
The solution to this issue is to break the non-stopping block of `buildTeams()` into smaller asynchronous blocks. When a lot of teams are to build, `buildTeams()` will build some of them (say 100 teams) and yield. Later, `buildTeams()` build another batch of teams.
This issue is related to Issue #2096.
Contributor guide
Research direction
Start by locating buildTeams() in the DD implementation and read how it currently builds all teams in one atomic block. Review Issue #2096 for related context and trace how asynchronous work can yield between batches. Done means large builds yield between bounded batches while preserving team creation and keeping DD responsive to heartbeats and other requests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100