apache / apache/pulsar

Reader skips over large block of messages after calling `seek`

Open
#6,847 10 comments 0 reactions 0 assignees View on GitHub
help wanted lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 22h
Merged PRs (30d)
142

Description

# Bug Description
In some difficult-to-reproduce circumstances, a reader will skip forward to later in the topic with no indication that anything abnormal happened.

# To Reproduce
1. Publish to a topic for a while; high volume might help; e.g., tens of millions of messages at least
2. Choose two timestamps, `t1` and `t2`, with a decent window between them, that are both within the `publishTimestamp` history for the topic
3. Do the following dozens of times
1. Create a reader and call `.seek(t1)`
2. Read initial messages until you get one in between `t1` and `t2` (I would think the first message should qualify, but often it doesn't, and I don't know why)
3. Read _and count_ messages until you see one with a `publishTimestamp > t2`
4. Print/return the count
4. Compare the results across runs

# Expected behavior
The result should always be the same, assuming the publisher is setting `publishTimestamp` monotonically and has already passed `t2` at the beginning of the test.

# Additional context
I often see that during the run the consumer makes a large jump. Just recently I have noticed that the point it jumps to (the "jump target") increases over time, in a way that I think tracks the messages being published to the topic. I saw a very reliable increase until I stopped the producer, and then the increase stopped.

## Configurations I've Tested
- Broker versions 2.4.2, 2.5.1
- Producer version 2.3.0, 2.5.1
- Just one producer at a time
- Non-partitioned topics
- Batching enabled
- Python client, but mostly Java client
- Both reader and consumer
- Tiered storage; I've observed the issue on timespans that precede, follow, and straddle the ledger offloading boundary, as well as topics that haven't been offloaded at all yet

Contributor guide

Open the contributing guide

Research direction

Start with the numbered reproduction using a high-volume topic, repeated reader or consumer calls to seek(t1), and counted messages through t2; compare results while varying batching, client language, and tiered storage. Done means identifying why the jump occurs and making repeated runs return consistent counts without skipping messages.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.