prometheus / prometheus/prometheus
Reduce the impact of remote write resharding
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 66.1k
- Forks
- 10.8k
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 131
Description
Right now resharding, especially sharding up is very disruptive to throughput. The resharding process drains all queues, which takes a significant amount of time if the remote endpoint is having issues. This will block new samples from being appended while they queues clear, and one slow shard can cause throughput to drop significantly.
Instead of waiting for all shards to flush to remote storage we could send them into the new shards that are being created, being sure to rebalance them into the appropriate shard.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the remote-write resharding implementation and the queue-draining path described in the issue. Study how shards are created and samples are assigned, then define completion as preserving throughput during resharding while rebalancing samples into the new shards without waiting for every existing queue to flush.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100