dapr / dapr/components-contrib
Implement the Apache Pulsar Reader Interface as an alternate Consumer Interface
- Dominant language
- Go
- Stars
- 602
- Forks
- 580
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
## Describe the feature
In Pulsar, the "standard" [consumer interface](https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#consumers) involves using consumers to listen on [topics](https://pulsar.apache.org/docs/2.11.x/reference-terminology/#topic), process incoming messages, and finally acknowledge those messages when they are processed. Whenever a new subscription is created, it is initially positioned at the end of the topic (by default), and consumers associated with that subscription begin reading with the first message created afterward. Whenever a consumer connects to a topic using a pre-existing subscription, it begins reading from the earliest message un-acknowledged within that subscription. In summary, with the consumer interface, subscription cursors are automatically managed by Pulsar in response to [message acknowledgments](https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#acknowledgment).
The reader interface for Pulsar enables applications to manually manage cursors. When you use a reader to connect to a topic---rather than a consumer---you need to specify which message the reader begins reading from when it connects to a topic. When connecting to a topic, the reader interface enables you to begin with:
The earliest available message in the topic.
The latest available message in the topic.
REMOVED FROM THIS REQUEST: [Some other messages between the earliest and the latest. If you select this option, you'll need to explicitly provide a message ID. Your application will be responsible for "knowing" this message ID in advance, perhaps fetching it from a persistent data store or cache.]
The reader interface is helpful for use cases like using Pulsar to provide effectively-once processing semantics for a stream processing system. For this use case, the stream processing system must be able to "rewind" topics to a specific message and begin reading there. The reader interface provides Pulsar clients with the low-level abstraction necessary to "manually position" themselves within a topic.
Internally, the reader interface is implemented as a consumer using an exclusive, non-durable subscription to the topic with a randomly-allocated name.
( Source: [Pulsar Concepts and Architecture - Pulsar Clients](https://pulsar.apache.org/docs/2.11.x/concepts-clients/) )
The Pulsar Reader Interface is very similar to that of a Consumer, but additional parameters to indicate where to start reading from in the cursor: a) from the start of the topic, b) from the last message ID in the topic or c) from a message ID somewhere in between and the last message ID in the topic. A PubSub component with key/value parameters for a Reader Interface could handle the basic options a) and b). REMOVED FROM THIS REQUEST: [For use case c) we need a practical way for the Dapr application to pass that value to the Dapr sidecar. Depending on what the application is doing, the starting message ID may change in a very dynamic manner.]
## Release Note
RELEASE NOTE:
Contributor guide
Research direction
The payload names no files or tests. Start by locating the existing Apache Pulsar consumer component and its configuration, then compare its behavior with the linked Pulsar reader documentation. Done means the component supports reader-based consumption from the topic start and latest message, with documented configuration and coverage for the new behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100