[Bug] The first consumption did not consume from the latest message position
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
pulsar 2.9.2
### Minimal reproduce step
1、Create a subscription to topic qlm-flink and set the initial position to latest:
bin/pulsar-admin topics create-subscription persistent://qlm-test/qlm-ns/qlm-flink --messageId latest --subscription qlm-sub
2、Set the retention time of topic qlm-flink to 8 hours, and write data continuously
3、One day later, use the subscription "qlm-sub" to consume the topic for the first time "persistent://qlm-test/qlm-ns/qlm-flink"
4、You will see that the consumption task will start consumption from 8 hours ago, not from the latest message position.This has huge implications for real-time tasks
Maybe it's something to do with these codes:
https://cdn-us.imgs.moe/2023/08/08/64d2222a7e719.jpg
### What did you expect to see?
I hope that if latest is set when creating a subscription, no matter when the consumption starts for the first time, it should start consumption from the latest consumption position
### What did you see instead?
Consumed a lot of historical data
### Anything else?
I want to know the purpose of this design. At the same time, I also want to know how to solve the scenario of persistent subscription, how to start consumption from "latest"
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start with the listed pulsar-admin topics create-subscription command using --messageId latest, then reproduce the delayed first-consumption case with topic retention enabled. Trace how the initial subscription position interacts with retained messages; done means the first consumer starts at the latest message position rather than consuming retained historical data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100