influxdata / influxdata/telegraf
kafka_consumer: expose consumer group lag as internal metric
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
## Feature Request
### Proposal:
Add the lag of the consumer group specified in [[[inputs.kafka_consumer]]](https://github.com/influxdata/telegraf/blob/master/plugins/inputs/kafka_consumer/) into the telegraf [[[inputs.internal]]](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/internal) metrics.
### Current behavior:
The input can lag with no indication of this exposed.
### Desired behavior:
When `[[inputs.internal]]` is enabled, the plugin adds selfstat items for the consumer group lag (other metrics might also be useful to add at this point). Sample output:
```
internal_kafka_consumer,instance=xxxx,consumer_group=tg-0,partition=0 current_offset=x,log_end_offset=y,lag=z 1654079199000000000
```
### Use case:
When a kafka consumer drops behind, it can be hard to diagnose. Kafka's own API does not expose consumer group offset metrics (they're stored in the offsets topics) and one might resort to the CLI tools, e.g.
```kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-application
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
tg-0 metrics-hosepipe 0 4562002220 4562002452 232 Telegraf-41eef470-f8fc-402a-9e1f-41b50ac153ed /1.2.3.4 Telegraf
tg-0 metrics-hosepipe 1 4561999766 4561999985 219 Telegraf-41eef470-f8fc-402a-9e1f-41b50ac153ed /1.2.3.4 Telegraf
```
While calls to the above could be wrapped in a script and called from Telegraf, the consumer input itself is in a better position to collect these metrics in context, apply tags etc.
Contributor guide
Research direction
Start by reading plugins/inputs/kafka_consumer/ and plugins/inputs/internal/ to understand how consumer data and internal selfstat metrics are collected. Implement consumer-group lag metrics with the proposed consumer_group and partition tags, then verify output includes current_offset, log_end_offset, and lag when inputs.internal is enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- observability, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100