linkedin / linkedin/Burrow

Not able to get lag or status info from endpoint on existing consumers

Open
#389 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

Hi, I'm trying to monitor Kafka consumers on our staging machine, below is the config file:

`
[general]
pidfile="./burrow.pid"
stdout-logfile="burrow.out"

[logging]
filename="log/burrow.log"
level="debug"
maxsize=100
maxbackups=30
maxage=10
use-localtime=false
use-compression=true

[zookeeper]
servers=[ "example.com:2181" ]
timeout=6

[client-profile.stage]
kafka-version="0.10.2"

[client-profile.production]
kafka-version="0.10.2"

[cluster.stage]
class-name="kafka"
servers=[ "example.com:9092" ]
client-profile="stage"
topic-refresh=120
offset-refresh=30
zookeeper-offset=true

[consumer.stage]
class-name="kafka"
cluster="stage"
servers=[ "example.com:9092" ]
client-profile="stage"
offsets-topic="__consumer_offsets"
#group-blacklist="^(console-consumer-|python-kafka-consumer-|quick-).*$"
group-blacklist=".*"
group-whitelist=""

[httpserver.default]
address=":18002"
`

the thing is I am able to see all the consumers using the corresponding endpoint but can't get the lag on the existing consumers, below is a snippet of the log:
`
{"level":"debug","ts":1524773696.0555763,"msg":"dropped","type":"module","coordinator":"storage","class":"inmemory","name":"default","worker":7,"cluster":"stage","consumer":"ca_raw","topic":"ca_raw","partition":1,"topic_partition_count":0,"offset":24,"timestamp":1484519101275,"owner":"","request":"StorageSetConsumerOffset","reason":"old offset"}`

also this is the result I got from hitting the /lag endpoint:

```
{
"error":
false,
"message":
"consumer status returned",
"request":
{
"host":
"MacBook-Pro.local",
"url":
"/v3/kafka/stage/consumer/ca_raw/lag"
},
"status":
{
"cluster":
"stage",
"complete":
1,
"group":
"ca_raw",
"maxlag":
null,
"partition_count":
0,
"partitions":
[],
"status":
"NOTFOUND",
"totallag":
0
}
}
```

Let me know if you need more information, thx in advance!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the `/v3/kafka/stage/consumer/ca_raw/lag` endpoint and the supplied Burrow configuration, then trace the `StorageSetConsumerOffset` log entry showing `topic_partition_count: 0` and `reason: "old offset"`. Reproduce the request against the existing consumer and determine why it returns `NOTFOUND`; done means the endpoint reports the consumer's lag or the missing prerequisite is clearly identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
observability-sre, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.