Collect metrics of Kafka consumer groups without active members
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 370
Description
`consumergroup` metricset of the Metricbeat `kafka` module collects consumer group metrics per member in the consumer group. When a consumer group doesn't have any active member, it doesn't collect any data, but this can be important to detect that all members are down, and to keep visualizing the last processed offsets.
Metricbeat should be able to collect metrics both, per consumergroup and per member.
I see two options for that:
* Split the consumergroup metricset in two, one for consumergroups, and another one for members. The one for members should take care of collecting the consumer lag for each member.
* In current consumergroup, generate an event for consumergroups that don't have any member, without the member-specific fields.
Example consumer group without members:
```
# ./kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --command-config /kafka/bin/sasl-consumer.properties --describe --group foo-1
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
Consumer group 'foo-1' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
foo 0 1 2 1 - - -
foo 1 1 3 2 - - -
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Metricbeat kafka module's consumergroup metricset and reproduce the no-active-members output shown in the issue. Compare the two proposed collection designs, then verify that consumer-group metrics are emitted without active members while member-specific lag remains available where applicable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100