apache / apache/pulsar-client-go

Enable accessing Message in TableView

Open
#1,176 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
745
Forks
389
Avg merge
3d 20h
Merged PRs (30d)
3

Description

**Is your feature request related to a problem? Please describe.**
I have a use case for processing messages in a TableView where I need to be able to access the metadata (e.g. publication time) for each message, for example to check if the message for a key is "out-of-date" or recent.

Currently this is not possible as the TableView implementation only provides the message as encoded by the SchemaValueType and discards all metadata.

**Describe the solution you'd like**
I would like to introduce two new methods to the TableView interface for accessing the associated `pulsar.Message` for each key:
1. `Message(key string) Message` - access the `pulsar.Message` for a given key (as an alternative to `Get`)
2.`Messages() map[string]Message` - access all the stored `pulsar.Message` objects for each key (as an alternative to `Entries`)

I propose to do this by changing how the messages are processed - instead of storing the encoded payload in `processMessage`, instead we store a map of map[string]Message. Backwards compatibility will be maintained by always returning the payload in the existing methods.

**Describe alternatives you've considered**
Alternatives include:
There are no current alternatives for accessing message metadata while using TableView.

Instead of changing any of the storage implementation, we can add a new map which additionally stores a mapping from key to `pulsar.Message`, and only use this map for the two new methods. I initially decided against this to avoid duplicating the logic for storing messages.

**Additional context**
NA.

Please let me know if this requires any further design or improvement documentation, happy to provide more detail if needed.

Contributor guide

Open the contributing guide

Research direction

Locate the TableView interface and the processMessage path, then inspect how the existing Get and Entries methods store and return payloads. Determine how the requested Message and Messages methods can preserve those existing behaviors while exposing message metadata; done means both new access paths work without breaking the current methods.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.