Burrow stops consuming from a single partition after segment roll
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
Hey,
We're experiencing issues with consuming messages from topic __consumer_offsets.
Sometimes when a segment roll for a partition happens on a broker, burrow stops consuming from that partition without any errors/warns/info.
Result - burrow shows "fake" lag for consumer groups which data was pushed to this particular partition.
So far we experienced it happening only on 2 partitions (12 and 13)
Example:
log from kafka broker:
`[2020-07-24 04:40:30,911] INFO [Log partition=__consumer_offsets-13, dir=/data_disk_0/kafka-logs] Rolled new log segment at offset 2375038040 in 2 ms. (kafka.log.Log)`
and prometheus showing lag:

We scheduled a cron job every hour to restart burrow for now. That's why lag disappears at 05:00 on the histogram above.
Topic details:
replication | 3
partitions | 50
segment.bytes | 104857600
compression.type | producer
cleanup.policy | compact
Burrow version 1.3.4
Kafka brokers version 2.1.0
Burrow config:
```
[storage.inmemory]
class-name="inmemory"
workers=20
[zookeeper]
servers=[ ]
timeout=6
root-path="/burrow"
[cluster.kafka-p-vm]
class-name="kafka"
servers=[ ]
[consumer.kafka-p-vm]
class-name="kafka"
cluster="kafka-p-vm"
servers=[ ]
start-latest=true
group-blacklist=""
group-whitelist=""
[httpserver.default]
address=":8000"
[client-profile.default-client]
kafka-version="2.1.0"
```
Broker config:
[Kafka broker config](https://gist.github.com/plelukas/9491583b8a8a2ca84ebbd3079243a490)
PS. I don't know if it's relevant but those are 2 biggest partitions in __consumer_offsets:
21G /data_disk_0/kafka-logs/__consumer_offsets-13
217M /data_disk_0/kafka-logs/__consumer_offsets-12
with the next biggest one to follow with:
99M /data_disk_0/kafka-logs/__consumer_offsets-47
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing a segment roll for __consumer_offsets partition 12 or 13 with Kafka 2.1.0 and Burrow 1.3.4, then compare consumption before and after the roll. Done means Burrow continues consuming the affected partition and reports current lag without requiring a restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- observability, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100