Graylog2 / Graylog2/graylog2-server
Add `client.rack` in kafka input with environment variables
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## What?
1. Add client.rack field in kafka input

2. read environment variable by default. or read from input setting
Let's say it is `GRAYLOG_RACK`. In that case, read `GRAYLOG_RACK` environment variable and set `client.rack=${GRAYLOG_RACK}` for kafka client consumer config
## Why?
If it is possible to let the client be rack aware (by setting `client.rack`), client can read from replica in same AZ. Let's say Kafka uses `broker.rack=us-east-1a` and `broker.rack=us=east-1c`. If we can configure `client.rack=us-east-1a` and `client.rack=us-east-1c`, we can save network cost. (AWS network cost within AZ is free)
We can configure `client.rack` in `Custom Kafka properties` field but should set manually for each node and it is too painful if node count is pretty high. Also, if node is running on Kubernetes, it can be recreated or scale-out/in so it is not the proper solution.
Reference: https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
## Your Environment
* Graylog Version: 4.2.6
* Elasticsearch Version:
* MongoDB Version:
* Operating System:
* Browser version:
Contributor guide
Assessment
This issue has not been assessed yet.