RealtimeToOfflineSegmentsTask does not honor maxNumRecordsPerTask
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
`RealtimeToOfflineSegmentsTask` does not honor `maxNumRecordsPerTask`, so it will generate only one sub-task per time bucket. In a table with high ingestion rate, each time bucket may contain >100M records, and using one single thread to process it can take very long time. We should enhance `RealtimeToOfflineSegmentsTask` to honor `maxNumRecordsPerTask` and be able to split one time bucket into multiple sub-tasks.
See #8014 for more details
Contributor guide
Research direction
Start by locating RealtimeToOfflineSegmentsTask and reviewing issue #8014 for the intended behavior. Trace how time buckets currently create sub-tasks and verify that a bucket with more than maxNumRecordsPerTask records is split into multiple sub-tasks, with high-ingestion cases covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100