Prometheus Metrics not showing all consumer groups
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
I'm running burrow (built from master as of June 3rd, 2020) against kafka 2.4.1. Prometheus metrics showing up in /metrics in but burrow_kafka_consumer_current_offset and burrow_kafka_consumer_partition_lag only show some of the consumer groups.
The missing consumer groups do appear in the /v3/kafka//consumer/ endpoint. But I've noticed that the missing groups have null for some partition offsets. i.e.
```
{
"offsets": [
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null
],
"owner": "/10.0.0.1",
"client_id": "consumer-8",
"current-lag": 0
},
```
The corresponding /v3/kafka//consumer//lag endpoint shows entries like:
```
{
"topic": "myTopic",
"partition": 2,
"owner": "/10.0.0.1",
"client_id": "consumer-8",
"status": "OK",
"start": null,
"end": null,
"current_lag": 0,
"complete": 0.06666667
}
```
I am currently assuming that the null's are due to this cluster being in our TST environment and our messages are produced keyed so it's not unusual for some partitions to see no traffic for days/weeks at a time which means it's possible that some offsets are past the retention period.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the /metrics output with the /v3/kafka//consumer/ and /lag endpoints for groups whose partition offsets are null. Trace how those endpoint results are represented in the Prometheus metrics, then define and test the expected handling so all relevant consumer groups are represented consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, prometheus
- Domain
- observability, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100