Reader skips over large block of messages after calling `seek`
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 142
Description
# Bug Description
In some difficult-to-reproduce circumstances, a reader will skip forward to later in the topic with no indication that anything abnormal happened.
# To Reproduce
1. Publish to a topic for a while; high volume might help; e.g., tens of millions of messages at least
2. Choose two timestamps, `t1` and `t2`, with a decent window between them, that are both within the `publishTimestamp` history for the topic
3. Do the following dozens of times
1. Create a reader and call `.seek(t1)`
2. Read initial messages until you get one in between `t1` and `t2` (I would think the first message should qualify, but often it doesn't, and I don't know why)
3. Read _and count_ messages until you see one with a `publishTimestamp > t2`
4. Print/return the count
4. Compare the results across runs
# Expected behavior
The result should always be the same, assuming the publisher is setting `publishTimestamp` monotonically and has already passed `t2` at the beginning of the test.
# Additional context
I often see that during the run the consumer makes a large jump. Just recently I have noticed that the point it jumps to (the "jump target") increases over time, in a way that I think tracks the messages being published to the topic. I saw a very reliable increase until I stopped the producer, and then the increase stopped.
## Configurations I've Tested
- Broker versions 2.4.2, 2.5.1
- Producer version 2.3.0, 2.5.1
- Just one producer at a time
- Non-partitioned topics
- Batching enabled
- Python client, but mostly Java client
- Both reader and consumer
- Tiered storage; I've observed the issue on timespans that precede, follow, and straddle the ledger offloading boundary, as well as topics that haven't been offloaded at all yet
Contributor guide
Research direction
Start with the numbered reproduction using a high-volume topic, repeated reader or consumer calls to seek(t1), and counted messages through t2; compare results while varying batching, client language, and tiered storage. Done means identifying why the jump occurs and making repeated runs return consistent counts without skipping messages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100