linkedin / linkedin/Burrow

Burrow stops consuming from a single partition after segment roll

Open
#649 0 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

Hey,

We're experiencing issues with consuming messages from topic __consumer_offsets.
Sometimes when a segment roll for a partition happens on a broker, burrow stops consuming from that partition without any errors/warns/info.
Result - burrow shows "fake" lag for consumer groups which data was pushed to this particular partition.
So far we experienced it happening only on 2 partitions (12 and 13)

Example:
log from kafka broker:
`[2020-07-24 04:40:30,911] INFO [Log partition=__consumer_offsets-13, dir=/data_disk_0/kafka-logs] Rolled new log segment at offset 2375038040 in 2 ms. (kafka.log.Log)`
and prometheus showing lag:
![image](https://user-images.githubusercontent.com/11867339/88385592-e9477780-cdae-11ea-8006-834127f5ead0.png)

We scheduled a cron job every hour to restart burrow for now. That's why lag disappears at 05:00 on the histogram above.

Topic details:
replication | 3
partitions | 50
segment.bytes | 104857600
compression.type | producer
cleanup.policy | compact

Burrow version 1.3.4
Kafka brokers version 2.1.0

Burrow config:
```
[storage.inmemory]
class-name="inmemory"
workers=20

[zookeeper]
servers=[ ]
timeout=6
root-path="/burrow"

[cluster.kafka-p-vm]
class-name="kafka"
servers=[ ]

[consumer.kafka-p-vm]
class-name="kafka"
cluster="kafka-p-vm"
servers=[ ]
start-latest=true
group-blacklist=""
group-whitelist=""

[httpserver.default]
address=":8000"

[client-profile.default-client]
kafka-version="2.1.0"
```

Broker config:
[Kafka broker config](https://gist.github.com/plelukas/9491583b8a8a2ca84ebbd3079243a490)

PS. I don't know if it's relevant but those are 2 biggest partitions in __consumer_offsets:
21G /data_disk_0/kafka-logs/__consumer_offsets-13
217M /data_disk_0/kafka-logs/__consumer_offsets-12

with the next biggest one to follow with:
99M /data_disk_0/kafka-logs/__consumer_offsets-47

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing a segment roll for __consumer_offsets partition 12 or 13 with Kafka 2.1.0 and Burrow 1.3.4, then compare consumption before and after the roll. Done means Burrow continues consuming the affected partition and reports current lag without requiring a restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
observability, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.