apache / apache/pinot

PEP: Redesign multi topic realtime ingestion support

Open
#18,739 0 comments 3 reactions 0 assignees View on GitHub
PEP-Request real-time
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Multitopic realtime ingestion as it currently exists has some issues. In this [PR](https://github.com/apache/pinot/pull/18433), @Jackie-Jiang said

The multi-topic partition id overloading existing partition id has caused so many issues:

We are missing checks here and there
Stream with custom partition id fails (See https://github.com/apache/pinot/issues/17260)
I think it is the time to revisit this design decision and avoid the troubles going forward. I can see 2 possible solutions:

Make partition_id part within the table name x_y where x is the topic id and y is the partition id. For single topic name, it remains the same
Leverage logical table for multiple topics, each topic a separate physical table

This is the PEP request to make those changes. I've detailed both suggestions in this doc:
https://docs.google.com/document/d/1dY6ZvaG_-FbTAQ5e1lxB4uvi8VGbR_WfCfCl5zD76A4/

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked Google design document and PR #18433, including the cited issue about custom partition IDs. Compare the two proposed redesigns and determine the required scope; the issue names no files, tests, or concrete completion criteria, so implementation readiness is not established.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.