Kafka lag increases for a topic but lag status is "OK"
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
Issue occurs on a Kafka Burrow v1.0.0, running on Ubuntu 16.04.4 LTS.
During some hours, lag is increasing on a topic for a consumer group consuming many other topics (no lag on the other topics).
See below the Burrow lag HTTP endpoint for the topic _logjson[0]_, each command is repeated at 1 minute interval.
```
# curl -s localhost:8080/v3/kafka/cluster/consumer/myconsumer/lag | jq '.status.partitions[] | select(.topic=="logjson") | select(.partition==0)'
{
"topic": "logjson",
"partition": 0,
"owner": "coll02/127.0.0.1",
"status": "OK",
"start": {
"offset": 6049200188,
"timestamp": 1539356997287,
"lag": 204894
},
"end": {
"offset": 6049227188,
"timestamp": 1539357030744,
"lag": 207963
},
"current_lag": 207963,
"complete": 1
}
# curl -s localhost:8080/v3/kafka/cluster/consumer/myconsumer/lag | jq '.status.partitions[] | select(.topic=="logjson") | select(.partition==0)'
{
"topic": "logjson",
"partition": 0,
"owner": "coll02/127.0.0.1",
"status": "OK",
"start": {
"offset": 6049245188,
"timestamp": 1539357048824,
"lag": 223394
},
"end": {
"offset": 6049272188,
"timestamp": 1539357086092,
"lag": 228876
},
"current_lag": 228876,
"complete": 1
}
# curl -s localhost:8080/v3/kafka/cluster/consumer/myconsumer/lag | jq '.status.partitions[] | select(.topic=="logjson") | select(.partition==0)'
{
"topic": "logjson",
"partition": 0,
"owner": "coll02/127.0.0.1",
"status": "OK",
"start": {
"offset": 6049296188,
"timestamp": 1539357112691,
"lag": 235911
},
"end": {
"offset": 6049323188,
"timestamp": 1539357145319,
"lag": 239049
},
"current_lag": 239049,
"complete": 1
}
# curl -s localhost:8080/v3/kafka/cluster/consumer/myconsumer/lag | jq '.status.partitions[] | select(.topic=="logjson") | select(.partition==0)'
{
"topic": "logjson",
"partition": 0,
"owner": "coll02/127.0.0.1",
"status": "OK",
"start": {
"offset": 6049347188,
"timestamp": 1539357172648,
"lag": 246101
},
"end": {
"offset": 6049374188,
"timestamp": 1539357206528,
"lag": 250063
},
"current_lag": 250063,
"complete": 1
}
# curl -s localhost:8080/v3/kafka/cluster/consumer/myconsumer/lag | jq '.status.partitions[] | select(.topic=="logjson") | select(.partition==0)'
{
"topic": "logjson",
"partition": 0,
"owner": "coll02/127.0.0.1",
"status": "OK",
"start": {
"offset": 6049398188,
"timestamp": 1539357235614,
"lag": 258183
},
"end": {
"offset": 6049425188,
"timestamp": 1539357268963,
"lag": 262250
},
"current_lag": 262250,
"complete": 1
}
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the report against Kafka Burrow v1.0.0 on Ubuntu 16.04.4 using the v3 consumer lag endpoint and repeated one-minute queries for the affected topic and partition. Trace how increasing lag is classified as OK; done means the reported status accurately reflects the observed lag progression.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100