Data distribution can fail to create teams on newly added hosts when many old hosts are excluded
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
Consider the case where we want to move a cluster to all new hosts. A common procedure for this is to add all of the new hosts to the cluster and then exclude all of the old hosts.
The data distributor will need to create new teams on the new hosts in order for data to be moved to these hosts. It chooses the maximum number of total "machine teams" that can be created based on the number of healthy machines in the cluster. Excluding all of the old machines can result in a significant decrease in the number of healthy machines and therefore also in the maximum number of machine teams.
Old machine teams still exist during this time, however, and count against the maximum. In the case where the number of fault domains has decreased between old and new hosts, the new maximum will be lower than the old maximum. As a result, it's possible that the number of old machine teams can exceed the new maximum and only 1 new team can be created.
With only 1 new team, data will be moved to only a small number of hosts, leaving the rests of the new hosts empty.
See the code [here](https://github.com/apple/foundationdb/blob/7506caaac4524a56248d44915d391de8c730b462/fdbserver/DataDistribution.actor.cpp#L2000) for where this computation is done.
Contributor guide
Assessment
This issue has not been assessed yet.