quickwit-oss / quickwit-oss/quickwit
Improve assignement of kafka source pipelines on several indexers
Open
@fulmicoton is already working on this.
Since Feb 26, 2024.
bug
enhancement
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
A user reported the following issue (I checked the cluster state):
- He has around 50 Kafka sources. The biggest Kafka source has 4 desired pipelines, most of which have one.
- He has 10 indexers, each with 4-6 pipelines. 1 indexer has all the pipelines of the biggest Kafka source.
This is bad because, as you can imagine, only 1 indexer is responsible for indexing the biggest Kafka source. And all other indexers are doing less work.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.