Preserve Kafka Message Metadata in Pinot Tables
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Opening this issue as per the conversation in Slack.
When streaming from Kafka, Pinot currently lacks of a way to allow users to uniquely identify messages. We may achieve it by exposing the metadata of Kafka messages such as offset/timestamp.
The approach suggested by @kishoreg is
> What we need is Kafka decoder to read this metadata and add it to generic row.
CC: @mayankshriv @npawar @xiangfu0
Contributor guide
Research direction
Start by locating the Kafka decoder and the generic row mentioned in the issue, then review the linked Slack discussion and existing comment thread for the intended metadata shape. The work is done when Kafka message metadata such as offset and timestamp is preserved and exposed in Pinot tables, with coverage for the resulting behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- data-engineering, databases, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100