apache / apache/pinot

Pinot creating segments again on restart and takes too much time

Open
#6,521 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Hi,

I am running pinot-ingestion-job with this params

`jobType: SegmentCreationAndTarPush
inputDirURI: 'inputdirectoryPath'
includeFileNamePattern: 'glob:**/*.csv.gz'
outputDirURI: 'ouputdirectoryPath'
overwriteOutput: false`

and everytime I restart pinot it created all the segment again for all the files and takes too much time for that too. Is there a way to persist the previously created segment and use those segments in next start to save time?

Contributor guide

Open the contributing guide

Research direction

Start by reading the pinot-ingestion-job documentation and the SegmentCreationAndTarPush job entry point, focusing on how inputDirURI, outputDirURI, and overwriteOutput are handled across restarts. Determine the expected persistence and reuse behavior for existing segments, then verify that restarting the job avoids recreating unchanged CSV.GZ inputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.