Pinot creating segments again on restart and takes too much time
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Hi,
I am running pinot-ingestion-job with this params
`jobType: SegmentCreationAndTarPush
inputDirURI: 'inputdirectoryPath'
includeFileNamePattern: 'glob:**/*.csv.gz'
outputDirURI: 'ouputdirectoryPath'
overwriteOutput: false`
and everytime I restart pinot it created all the segment again for all the files and takes too much time for that too. Is there a way to persist the previously created segment and use those segments in next start to save time?
Contributor guide
Research direction
Start by reading the pinot-ingestion-job documentation and the SegmentCreationAndTarPush job entry point, focusing on how inputDirURI, outputDirURI, and overwriteOutput are handled across restarts. Determine the expected persistence and reuse behavior for existing segments, then verify that restarting the job avoids recreating unchanged CSV.GZ inputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100