Split batch_aggregation_shard_count configuration in two
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 70
- Forks
- 16
- Avg merge
- 4h 36m
- Merged PRs (30d)
- 30
Description
We've been discussing increasing batch_aggregation_shard_count. I think it would be good to split this into two configuration parameters in order to make deploying such changes safer. We could use one count for writes to random shards, and another for broadcast writes to all shards, which advance the overall batch's state machine. (note that reads scan all available shards, so the batch_aggregation_shard_count configuration parameter does not directly impact the read path) So long as all running processes and all existing batches in the database use a count for broadcast writes that is greater than or equal to the count for randomly sharded writes, then our justification for the batch state machine is sound.
Firstly, this would provide a safe avenue for decreasing the shard count, by decreasing the count for random writes, waiting for all the data to turn over, and then decreasing the count for broadcast writes. Secondly, this would make changes to this parameter when running as the helper aggregator safer. Note that, for the leader, it would be possible to sequence an increase in batch_aggregation_shard_count between the different components to obey the above rule, i.e. collection job driver first. For the helper role, however, we do both random sharded writes and broadcast writes from the same process, so a rolling update of the configuration parameter as-is may introduce a small window for strange race conditions.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating references to batch_aggregation_shard_count and tracing the random-shard and broadcast-write paths described in the issue. The change is done when those paths use separate counts and the documented invariant keeps the broadcast-write count greater than or equal to the random-write count during deployment changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100