PEP: Redesign multi topic realtime ingestion support
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
Multitopic realtime ingestion as it currently exists has some issues. In this [PR](https://github.com/apache/pinot/pull/18433), @Jackie-Jiang said
The multi-topic partition id overloading existing partition id has caused so many issues:
We are missing checks here and there
Stream with custom partition id fails (See https://github.com/apache/pinot/issues/17260)
I think it is the time to revisit this design decision and avoid the troubles going forward. I can see 2 possible solutions:
Make partition_id part within the table name x_y where x is the topic id and y is the partition id. For single topic name, it remains the same
Leverage logical table for multiple topics, each topic a separate physical table
This is the PEP request to make those changes. I've detailed both suggestions in this doc:
https://docs.google.com/document/d/1dY6ZvaG_-FbTAQ5e1lxB4uvi8VGbR_WfCfCl5zD76A4/
Contributor guide
Research direction
Start by reading the linked Google design document and PR #18433, including the cited issue about custom partition IDs. Compare the two proposed redesigns and determine the required scope; the issue names no files, tests, or concrete completion criteria, so implementation readiness is not established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100