apache / apache/pinot

Preserve Kafka Message Metadata in Pinot Tables

Open
#7,004 32 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Opening this issue as per the conversation in Slack.

When streaming from Kafka, Pinot currently lacks of a way to allow users to uniquely identify messages. We may achieve it by exposing the metadata of Kafka messages such as offset/timestamp.

The approach suggested by @kishoreg is

> What we need is Kafka decoder to read this metadata and add it to generic row.

CC: @mayankshriv @npawar @xiangfu0

Contributor guide

Open the contributing guide

Research direction

Start by locating the Kafka decoder and the generic row mentioned in the issue, then review the linked Slack discussion and existing comment thread for the intended metadata shape. The work is done when Kafka message metadata such as offset and timestamp is preserved and exposed in Pinot tables, with coverage for the resulting behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, databases, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.