elastic / elastic/beats

Collect metrics of Kafka consumer groups without active members

Open
#23,077 9 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug Metricbeat module Team:Service-Integrations
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 1h
Merged PRs (30d)
370

Description

`consumergroup` metricset of the Metricbeat `kafka` module collects consumer group metrics per member in the consumer group. When a consumer group doesn't have any active member, it doesn't collect any data, but this can be important to detect that all members are down, and to keep visualizing the last processed offsets.

Metricbeat should be able to collect metrics both, per consumergroup and per member.

I see two options for that:
* Split the consumergroup metricset in two, one for consumergroups, and another one for members. The one for members should take care of collecting the consumer lag for each member.
* In current consumergroup, generate an event for consumergroups that don't have any member, without the member-specific fields.

Example consumer group without members:
```
# ./kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --command-config /kafka/bin/sasl-consumer.properties --describe --group foo-1
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
Consumer group 'foo-1' has no active members.

TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
foo 0 1 2 1 - - -
foo 1 1 3 2 - - -
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Metricbeat kafka module's consumergroup metricset and reproduce the no-active-members output shown in the issue. Compare the two proposed collection designs, then verify that consumer-group metrics are emitted without active members while member-specific lag remains available where applicable.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.