influxdata / influxdata/telegraf

Add a kafka_consumer_lag input to collect consumer group lag natively via the Kafka Admin API

Open
#19,736 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Use Case

Telegraf can produce to and consume from Kafka, but the single most common Kafka health metric — consumer group lag — currently requires deploying Burrow or kafka_exporter. This plugin computes lag directly using the existing common/kafka client configuration, with no external components.

### Expected behavior

1. Connects to the cluster
2. Lists consumer groups, fetches committed offsets per group
3. Emit one metric per (group, topic, partition)

A new input plugin, `inputs.kafka_consumer_lag`:
```
[[inputs.kafka_consumer_lag]]
## Kafka brokers
brokers = ["localhost:9092"]

## Consumer groups to monitor, glob patterns. Default: all groups.
# groups_include = ["*"]
# groups_exclude = []

## Topics to monitor, glob patterns. Default: all topics except internal ones.
# topics_include = ["*"]
# topics_exclude = []

## Emit per-partition metrics (kafka_consumer_lag).
# partition_metrics = true
## Emit per-topic aggregated metrics (kafka_consumer_lag_topic).
# topic_metrics = true

## Only collect groups whose coordinator is this broker ID. Intended for
## agent-per-broker deployments so each group is collected exactly once
## and collection fails over automatically with coordinator moves.
# coordinator_broker_id = -1

## Optional SASL / TLS settings, same as inputs.kafka_consumer
# sasl_username = ""
# sasl_password = ""
# sasl_mechanism = ""
# tls_ca = ""
# insecure_skip_verify = false
```

### Actual behavior

There is no Telegraf-only way to collect consumer group lag today. Users must deploy Burrow or a Prometheus exporter, or attach Jolokia to every consumer application. On a broker-side Jolokia setup the lag simply cannot be derived because the broker never exposes committed offsets over JMX.

### Additional info

I am willing to implement this and submit a PR.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing inputs.kafka_consumer plugin and the shared common/kafka client configuration. Define the new inputs.kafka_consumer_lag entry point around the listed group, topic, partition, and coordinator options. Done means the plugin connects through the Kafka Admin API and emits the requested per-partition and optional per-topic lag metrics without external components.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.