Fix large partition id for single stream
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
When multiple stream support was added in #13790, it assumes the partition id for a given stream can never go over `10000`. This assumption cannot be held for certain custom stream plugins.
#15957 fixed the problem by always checking if it is single stream before applying the modulo operation in `IngestionConfigUtils` (`getStreamPartitionIdFromPinotPartitionId` and `getStreamConfigIndexFromPinotPartitionId`).
Several recent PRs break this again, and we need to fix it:
- #16783 @noob-se7en
- #16833 @lnbest0707-uber
- #17163 @noob-se7en
- #17217 @tarun11Mavani
In the meanwhile, we should consider adding a test for this scenario.
@jadami10 Please confirm if you are still using a plugin that can generate large partition id
Contributor guide
Research direction
Start in IngestionConfigUtils at getStreamPartitionIdFromPinotPartitionId and getStreamConfigIndexFromPinotPartitionId, then inspect the recent changes referenced in the issue. Add a regression test for a single stream with a partition id over 10000; done means the partition id is preserved without the incorrect modulo behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100