apache / apache/pinot

Fix large partition id for single stream

Open
#17,260 10 comments 0 reactions 0 assignees View on GitHub
bug ingestion real-time
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

When multiple stream support was added in #13790, it assumes the partition id for a given stream can never go over `10000`. This assumption cannot be held for certain custom stream plugins.

#15957 fixed the problem by always checking if it is single stream before applying the modulo operation in `IngestionConfigUtils` (`getStreamPartitionIdFromPinotPartitionId` and `getStreamConfigIndexFromPinotPartitionId`).

Several recent PRs break this again, and we need to fix it:
- #16783 @noob-se7en
- #16833 @lnbest0707-uber
- #17163 @noob-se7en
- #17217 @tarun11Mavani

In the meanwhile, we should consider adding a test for this scenario.

@jadami10 Please confirm if you are still using a plugin that can generate large partition id

Contributor guide

Open the contributing guide

Research direction

Start in IngestionConfigUtils at getStreamPartitionIdFromPinotPartitionId and getStreamConfigIndexFromPinotPartitionId, then inspect the recent changes referenced in the issue. Add a regression test for a single stream with a partition id over 10000; done means the partition id is preserved without the incorrect modulo behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.