linkedin / linkedin/Burrow

No lag reported in metrics

Open
#689 2 comments 7 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

We are using Burrow, v1.3.4 against Amazon MSK (Apache Kafka), v2.5.1.
Trying to setup burrow/prometheus/grafana monitoring, we faced following issue:

### On stage env
*/metrics* endpoint of burrow shows everything that is expected regarding our custom topics, including:
* **burrow_kafka_consumer_current_offset**
* **burrow_kafka_consumer_partition_lag**
* **burrow_kafka_topic_partition_offset**

REST endpoints show the same, everything looks fine.

### On live env
**/metrics** endpoint shows values of:
* **burrow_kafka_consumer_current_offset** and **burrow_kafka_consumer_partition_lag** for ONLY some internal Amazon MSK topic (__amazon_msk_canary), **BUT NOT FOR ANY CUSTOM TOPIC OF OURS**
* **burrow_kafka_topic_partition_offset** for our custom topics as well as for ___amazon_msk_canary_

What's even more strange, REST endpoints show all as expected for all topics.

After spending a whole day on research (kafka upgrade from v2.2 to v2.5.1, restarting the Burrow, artificially creating the lag by turning off the consumer group) we've totally stuck.

Any help would be appreciated.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Compare Burrow's /metrics and REST endpoint results for custom topics in the reported stage and live environments, using Burrow v1.3.4 with Amazon MSK/Kafka v2.5.1. Start by checking the consumer offset and partition lag metrics against the topic partition offset metrics; done means identifying why live custom-topic consumer metrics are missing and documenting or verifying the resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, grafana, kafka, prometheus
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.