apache / apache/pulsar

Add ability to seek from current position in Reader interface

Open
#10,488 7 comments 2 reactions 0 assignees View on GitHub
lifecycle/stale type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 22h
Merged PRs (30d)
142

Description

**Is your enhancement request related to a problem? Please describe.**
The Reader interface has the ability to seek to an arbitrary timestamp, MessageId, first or last message in a topic, or to read the next message. However, there is no API to seek fowards/backwards from the current position in a topic.

Ours I'm sure is not a common use-case, but you might imagine other situations where this could be useful. Ours is that we originally used Kafka for our pub/sub system and implemented cluster-wide logging via Kafka's Log4jAppender. We tried to implement the same thing using Pulsar, but we are unable to request the last _n_ logs because we can't seek backwards _n_ messages from the end of the topic. (Requesting logs from a particular timestamp onward was much easier to implement than in Kafka though!)

**Describe the solution you'd like**
Add a method to seek backwards (and forwards) from the current position by a number of messages.

**Describe alternatives you've considered**
**Edit:** See the workaround in my comment below.
~~I tried finding the last available MessageId and then manipulating the fields. Decrementing the MessageIdImpl.entryId seems to work, but only to the beginning of the ledger. I could also decrement the ledgerId, but I wouldn't know the Ledger's top entryId to start with. Plus, I'm not sure this is always a safe thing to do!~~

Contributor guide

Open the contributing guide

Research direction

Start with the Reader interface and its existing seek APIs, then review the issue's comment thread, including the documented workaround. Define how seeking forward and backward by a message count should behave across topic boundaries before implementing it, and verify the behavior with the relevant Reader tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.