apache / apache/pulsar

[Bug] The first consumption did not consume from the latest message position

Open
#20,954 5 comments 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

pulsar 2.9.2

### Minimal reproduce step

1、Create a subscription to topic qlm-flink and set the initial position to latest:
bin/pulsar-admin topics create-subscription persistent://qlm-test/qlm-ns/qlm-flink --messageId latest --subscription qlm-sub

2、Set the retention time of topic qlm-flink to 8 hours, and write data continuously

3、One day later, use the subscription "qlm-sub" to consume the topic for the first time "persistent://qlm-test/qlm-ns/qlm-flink"

4、You will see that the consumption task will start consumption from 8 hours ago, not from the latest message position.This has huge implications for real-time tasks

Maybe it's something to do with these codes:
https://cdn-us.imgs.moe/2023/08/08/64d2222a7e719.jpg

### What did you expect to see?

I hope that if latest is set when creating a subscription, no matter when the consumption starts for the first time, it should start consumption from the latest consumption position

### What did you see instead?

Consumed a lot of historical data

### Anything else?

I want to know the purpose of this design. At the same time, I also want to know how to solve the scenario of persistent subscription, how to start consumption from "latest"

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the listed pulsar-admin topics create-subscription command using --messageId latest, then reproduce the delayed first-consumption case with topic retention enabled. Trace how the initial subscription position interacts with retained messages; done means the first consumer starts at the latest message position rather than consuming retained historical data.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.