apache / apache/pulsar

[Bug] key_shared consumer with backlog cause pulsar broker read a huge amount data from bookkeeper but dispatch a little, cause network bandwith exhausted

Open
#23,514 1 comment 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### Version

linux x64, java17, pulsar 3.0.4, pulsar c++ client 3.4.2

### Minimal reproduce step

1. stop key_shared consumer 5 mininutes
2. start key_shared consumer

### What did you expect to see?

1. broker the data dispatch to consumer is closely to the the data read speed from bookie
2. consumer can finish read backlog after a couple of minutes

### What did you see instead?

1. broker read a lot of data from bookkeeper only dispatch a little

the node with only broker: it‘s network receive is very big, but only transmit a little data to consumer
![Snipaste_2024-10-25_14-55-27](https://github.com/user-attachments/assets/95ddf802-4605-4aa2-97b4-11aeba7b156d)

the node with only bookeeper: it‘s network transmit is 23Gb/s, network band with reach the hardware limit ;

2. consumer can not catch up;

### Anything else?

the bookkeeper disk read is very low; so the data is in the bookkeeper read cache; I think maybe some bug cause pulsar broker keep read same entry from bookkeeper , cause the network bandwith exhausted

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue by stopping a key_shared consumer for five minutes and restarting it on the reported Pulsar 3.0.4 and C++ client 3.4.2 versions. Compare broker-to-bookkeeper reads with consumer dispatch and backlog recovery; done means the broker no longer reads substantially more data than it dispatches and the consumer catches up without exhausting network bandwidth.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.