influxdata / influxdata/telegraf

kafka_consumer: expose consumer group lag as internal metric

Open
#11,231 3 comments 0 reactions 0 assignees View on GitHub
area/kafka feature request
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

## Feature Request

### Proposal:

Add the lag of the consumer group specified in [[[inputs.kafka_consumer]]](https://github.com/influxdata/telegraf/blob/master/plugins/inputs/kafka_consumer/) into the telegraf [[[inputs.internal]]](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/internal) metrics.

### Current behavior:
The input can lag with no indication of this exposed.

### Desired behavior:
When `[[inputs.internal]]` is enabled, the plugin adds selfstat items for the consumer group lag (other metrics might also be useful to add at this point). Sample output:
```
internal_kafka_consumer,instance=xxxx,consumer_group=tg-0,partition=0 current_offset=x,log_end_offset=y,lag=z 1654079199000000000
```

### Use case:
When a kafka consumer drops behind, it can be hard to diagnose. Kafka's own API does not expose consumer group offset metrics (they're stored in the offsets topics) and one might resort to the CLI tools, e.g.
```kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-application
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
tg-0 metrics-hosepipe 0 4562002220 4562002452 232 Telegraf-41eef470-f8fc-402a-9e1f-41b50ac153ed /1.2.3.4 Telegraf
tg-0 metrics-hosepipe 1 4561999766 4561999985 219 Telegraf-41eef470-f8fc-402a-9e1f-41b50ac153ed /1.2.3.4 Telegraf
```
While calls to the above could be wrapped in a script and called from Telegraf, the consumer input itself is in a better position to collect these metrics in context, apply tags etc.

Contributor guide

Open the contributing guide

Research direction

Start by reading plugins/inputs/kafka_consumer/ and plugins/inputs/internal/ to understand how consumer data and internal selfstat metrics are collected. Implement consumer-group lag metrics with the proposed consumer_group and partition tags, then verify output includes current_offset, log_end_offset, and lag when inputs.internal is enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
observability, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.