linkedin / linkedin/Burrow

Lag summary per topic [question]

Open
#362 1 comment 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

Hello

Currently when querying for lag i receive information in form:
"status" -> "partitions" -> "partition topic".

When topic is located on multiple partitions i need to summarize (correct me if i am wrong) current_lag from all partition containing particular topic.
The issue that when gathering data using collectd topic is part of partition hash:

`{
"topic": "sometopic",
"partition": 1,
"owner": "",
"status": "OK",
"start": {
"offset": 25836605,
"timestamp": 1519389278989,
"lag": 0
},
"end": {
"offset": 25836701799,
"timestamp": 1519389368055,
"lag": 0
},
"current_lag": 190,
"complete": 1
},
`
making it impossible to sum current_lag value from only single topic.

Am i missing a simple way of getting summary lag per topic using ELK + collectd?

Contributor guide

No contributing guide indexed for this repository

Research direction

No source file or test is named. Start by reproducing the partition-level lag response described in the issue and checking how Burrow output is consumed by collectd and ELK. Done should be a confirmed answer about topic-level aggregation, with any required implementation or usage guidance explicitly identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, go, kafka
Domain
backend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.