divviup / divviup/janus

Split batch_aggregation_shard_count configuration in two

Open
#3,463 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
70
Forks
16
Avg merge
4h 36m
Merged PRs (30d)
30

Description

We've been discussing increasing batch_aggregation_shard_count. I think it would be good to split this into two configuration parameters in order to make deploying such changes safer. We could use one count for writes to random shards, and another for broadcast writes to all shards, which advance the overall batch's state machine. (note that reads scan all available shards, so the batch_aggregation_shard_count configuration parameter does not directly impact the read path) So long as all running processes and all existing batches in the database use a count for broadcast writes that is greater than or equal to the count for randomly sharded writes, then our justification for the batch state machine is sound.

Firstly, this would provide a safe avenue for decreasing the shard count, by decreasing the count for random writes, waiting for all the data to turn over, and then decreasing the count for broadcast writes. Secondly, this would make changes to this parameter when running as the helper aggregator safer. Note that, for the leader, it would be possible to sequence an increase in batch_aggregation_shard_count between the different components to obey the above rule, i.e. collection job driver first. For the helper role, however, we do both random sharded writes and broadcast writes from the same process, so a rolling update of the configuration parameter as-is may introduce a small window for strange race conditions.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating references to batch_aggregation_shard_count and tracing the random-shard and broadcast-write paths described in the issue. The change is done when those paths use separate counts and the documented invariant keeps the broadcast-write count greater than or equal to the random-write count during deployment changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.