influxdata / influxdata/telegraf
Add a kafka_consumer_lag input to collect consumer group lag natively via the Kafka Admin API
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Use Case
Telegraf can produce to and consume from Kafka, but the single most common Kafka health metric — consumer group lag — currently requires deploying Burrow or kafka_exporter. This plugin computes lag directly using the existing common/kafka client configuration, with no external components.
### Expected behavior
1. Connects to the cluster
2. Lists consumer groups, fetches committed offsets per group
3. Emit one metric per (group, topic, partition)
A new input plugin, `inputs.kafka_consumer_lag`:
```
[[inputs.kafka_consumer_lag]]
## Kafka brokers
brokers = ["localhost:9092"]
## Consumer groups to monitor, glob patterns. Default: all groups.
# groups_include = ["*"]
# groups_exclude = []
## Topics to monitor, glob patterns. Default: all topics except internal ones.
# topics_include = ["*"]
# topics_exclude = []
## Emit per-partition metrics (kafka_consumer_lag).
# partition_metrics = true
## Emit per-topic aggregated metrics (kafka_consumer_lag_topic).
# topic_metrics = true
## Only collect groups whose coordinator is this broker ID. Intended for
## agent-per-broker deployments so each group is collected exactly once
## and collection fails over automatically with coordinator moves.
# coordinator_broker_id = -1
## Optional SASL / TLS settings, same as inputs.kafka_consumer
# sasl_username = ""
# sasl_password = ""
# sasl_mechanism = ""
# tls_ca = ""
# insecure_skip_verify = false
```
### Actual behavior
There is no Telegraf-only way to collect consumer group lag today. Users must deploy Burrow or a Prometheus exporter, or attach Jolokia to every consumer application. On a broker-side Jolokia setup the lag simply cannot be derived because the broker never exposes committed offsets over JMX.
### Additional info
I am willing to implement this and submit a PR.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing inputs.kafka_consumer plugin and the shared common/kafka client configuration. Define the new inputs.kafka_consumer_lag entry point around the listed group, topic, partition, and coordinator options. Done means the plugin connects through the Kafka Admin API and emits the requested per-partition and optional per-topic lag metrics without external components.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100