linkedin / linkedin/Burrow

Prometheus Metrics not showing all consumer groups

Open
#637 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

I'm running burrow (built from master as of June 3rd, 2020) against kafka 2.4.1. Prometheus metrics showing up in /metrics in but burrow_kafka_consumer_current_offset and burrow_kafka_consumer_partition_lag only show some of the consumer groups.

The missing consumer groups do appear in the /v3/kafka//consumer/ endpoint. But I've noticed that the missing groups have null for some partition offsets. i.e.

```
{
"offsets": [
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null
],
"owner": "/10.0.0.1",
"client_id": "consumer-8",
"current-lag": 0
},
```

The corresponding /v3/kafka//consumer//lag endpoint shows entries like:

```
{
"topic": "myTopic",
"partition": 2,
"owner": "/10.0.0.1",
"client_id": "consumer-8",
"status": "OK",
"start": null,
"end": null,
"current_lag": 0,
"complete": 0.06666667
}
```

I am currently assuming that the null's are due to this cluster being in our TST environment and our messages are produced keyed so it's not unusual for some partitions to see no traffic for days/weeks at a time which means it's possible that some offsets are past the retention period.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the /metrics output with the /v3/kafka//consumer/ and /lag endpoints for groups whose partition offsets are null. Trace how those endpoint results are represented in the Prometheus metrics, then define and test the expected handling so all relevant consumer groups are represented consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, prometheus
Domain
observability, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.