quickwit-oss / quickwit-oss/quickwit
Add Kafka rack awareness/Fetch from follower to Kafka source configuration
@dayaffe is already working on this.
Since Aug 28, 2026.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
This feature will make Kafka cross availability zone traffic in any cloud hosting provider free of charge. This is a huge bill than you process a lot of logs and your Kafka infrastructure is spread between multiple availability zones.
LibrdKafka library does support this as per https://docs.confluent.io/platform/current/clients/librdkafka/html/md_INTRODUCTION.html#fetch-from-follower , but we can't use this configuration, because QW assigns Kafka pipelines on its own and we can't just hardcoded Kafka client rack parameter in Kafka Source configuration like this:
version: 0.8
source_id: my-kafka-source
source_type: kafka
num_pipelines: 2
params:
topic: my-topic
client_params:
bootstrap.servers: localhost:9092
client.rack:us-east-1a
Kafka source configuration should support fetching client.rack from the environment variable on the QW indexer server.
For example you can use the following configuration of the source:
version: 0.8
source_id: my-kafka-source
source_type: kafka
num_pipelines: 2
params:
topic: my-topic
client_params:
bootstrap.servers: localhost:9092
client.rack:${QW_KAFKA_CLIENT_RACK}
Now when pipeline is scheduled on QW Indexer with env variable QW_KAFKA_CLIENT_RACK set to "us-east-1a" Kafka source config will take it from that variable and set as a client.rack parameter during connection to the Kafka server.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.