Graylog2 / Graylog2/graylog2-server

Add `client.rack` in kafka input with environment variables

Open
#12,196 0 comments 0 reactions 0 assignees View on GitHub
feature inputs triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## What?

1. Add client.rack field in kafka input

![image](https://user-images.githubusercontent.com/6358016/155772451-a91ad6ef-4a60-4d10-87ed-43be10c01769.png)

2. read environment variable by default. or read from input setting

Let's say it is `GRAYLOG_RACK`. In that case, read `GRAYLOG_RACK` environment variable and set `client.rack=${GRAYLOG_RACK}` for kafka client consumer config

## Why?

If it is possible to let the client be rack aware (by setting `client.rack`), client can read from replica in same AZ. Let's say Kafka uses `broker.rack=us-east-1a` and `broker.rack=us=east-1c`. If we can configure `client.rack=us-east-1a` and `client.rack=us-east-1c`, we can save network cost. (AWS network cost within AZ is free)

We can configure `client.rack` in `Custom Kafka properties` field but should set manually for each node and it is too painful if node count is pretty high. Also, if node is running on Kubernetes, it can be recreated or scale-out/in so it is not the proper solution.

Reference: https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica

## Your Environment

* Graylog Version: 4.2.6
* Elasticsearch Version:
* MongoDB Version:
* Operating System:
* Browser version:

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.