KeyShared consumers with sticky hash range generate backlog with messages not fall within consumer hash range - v2.9.2
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Describe the bug**
KeyShared consumers with sticky hash range generate backlog with messages not fall within consumer hash range, and that cause consumers to not receive any messages even for the ones that fall in its range .
**To Reproduce**
Steps to reproduce the behavior:
1. create Keyshared consumer with sticky has range example 35891,35899
2. create producer to generate message with Murmur3_32Hash hashing schema , and the key hash value is outside the consumer range , ex : 35770
3. Consumer will not receive message as expected , however , the message remains in the backlog
4. Now , let producer generate message with key hash fall within the consumer range
5. Consumer still not received the message
Note : IF consumer disconnect and reconnect , it will receive the expected message , however , the same scenario will be repeated for any new messages as long as the consumer has backlog for messages that does not fall within its range
**Expected behavior**
Consumer should have backlog for only messages that fall within its range , and should be able to consume that message
**Desktop (please complete the following information):**
OS: Ubuntu 18.04
Java 8
Pulsar running as systemd service
6 brokers
6 bookies
5 ZK
Additional context
Pulsar v2.9.2
No k8 deployment , and client accessing brokers DNS directly without Pulsar proxy.
Contributor guide
Research direction
Start by reproducing the KeyShared sticky hash range scenario with an out-of-range key followed by an in-range key, using the Murmur3_32Hash schema. Trace the consumer backlog and hash-range assignment behavior; done means out-of-range messages no longer block delivery of messages within the consumer's range, including after reconnects.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100