linkedin / linkedin/Burrow

Kafka lag increases for a topic but lag status is "OK"

Open
#453 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

Issue occurs on a Kafka Burrow v1.0.0, running on Ubuntu 16.04.4 LTS.

During some hours, lag is increasing on a topic for a consumer group consuming many other topics (no lag on the other topics).

See below the Burrow lag HTTP endpoint for the topic _logjson[0]_, each command is repeated at 1 minute interval.

```
# curl -s localhost:8080/v3/kafka/cluster/consumer/myconsumer/lag | jq '.status.partitions[] | select(.topic=="logjson") | select(.partition==0)'
{
"topic": "logjson",
"partition": 0,
"owner": "coll02/127.0.0.1",
"status": "OK",
"start": {
"offset": 6049200188,
"timestamp": 1539356997287,
"lag": 204894
},
"end": {
"offset": 6049227188,
"timestamp": 1539357030744,
"lag": 207963
},
"current_lag": 207963,
"complete": 1
}
# curl -s localhost:8080/v3/kafka/cluster/consumer/myconsumer/lag | jq '.status.partitions[] | select(.topic=="logjson") | select(.partition==0)'
{
"topic": "logjson",
"partition": 0,
"owner": "coll02/127.0.0.1",
"status": "OK",
"start": {
"offset": 6049245188,
"timestamp": 1539357048824,
"lag": 223394
},
"end": {
"offset": 6049272188,
"timestamp": 1539357086092,
"lag": 228876
},
"current_lag": 228876,
"complete": 1
}
# curl -s localhost:8080/v3/kafka/cluster/consumer/myconsumer/lag | jq '.status.partitions[] | select(.topic=="logjson") | select(.partition==0)'
{
"topic": "logjson",
"partition": 0,
"owner": "coll02/127.0.0.1",
"status": "OK",
"start": {
"offset": 6049296188,
"timestamp": 1539357112691,
"lag": 235911
},
"end": {
"offset": 6049323188,
"timestamp": 1539357145319,
"lag": 239049
},
"current_lag": 239049,
"complete": 1
}
# curl -s localhost:8080/v3/kafka/cluster/consumer/myconsumer/lag | jq '.status.partitions[] | select(.topic=="logjson") | select(.partition==0)'
{
"topic": "logjson",
"partition": 0,
"owner": "coll02/127.0.0.1",
"status": "OK",
"start": {
"offset": 6049347188,
"timestamp": 1539357172648,
"lag": 246101
},
"end": {
"offset": 6049374188,
"timestamp": 1539357206528,
"lag": 250063
},
"current_lag": 250063,
"complete": 1
}
# curl -s localhost:8080/v3/kafka/cluster/consumer/myconsumer/lag | jq '.status.partitions[] | select(.topic=="logjson") | select(.partition==0)'
{
"topic": "logjson",
"partition": 0,
"owner": "coll02/127.0.0.1",
"status": "OK",
"start": {
"offset": 6049398188,
"timestamp": 1539357235614,
"lag": 258183
},
"end": {
"offset": 6049425188,
"timestamp": 1539357268963,
"lag": 262250
},
"current_lag": 262250,
"complete": 1
}

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the report against Kafka Burrow v1.0.0 on Ubuntu 16.04.4 using the v3 consumer lag endpoint and repeated one-minute queries for the affected topic and partition. Trace how increasing lag is classified as OK; done means the reported status accurately reflects the observed lag progression.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.