apache / apache/pinot

Unable to upload existing segments to a table

Open
#19,384 3 comments 0 reactions 0 assignees View on GitHub
ingestion troubleshooting
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Hello,

I try to upload some old segments produced by an REALTIME table to an OFFLINE table.

It is working as expected using the API as explained here: https://docs.pinot.apache.org/build-with-pinot/ingestion/batch-ingestion/segment-upload#uri-push

But the API is for 1 segment and I want to upload an entire "directory" with many segments. For this a job using Pinot Admin seems to be the solution.
I tried using "SegmentUriPush" but it does not work because my segment are named "myTable__0__0__20260601T1557Z" with no tar.gz extension.
But this extension seems to be required when the job is used (this is not the case using the API).

Am I doing something wrong or is it a limitation that can be fixed in a future version ?

I use Azure Data Lake Storage as a deep store.

Thanks for your help.

Contributor guide

Open the contributing guide

Research direction

Start with the SegmentUriPush job and compare its handling of segment names with the documented API URI-push behavior, especially for Azure Data Lake Storage. Confirm whether extensionless segments from the named directory are expected to upload; done means the job handles these segments consistently or the limitation is clearly established.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, java
Domain
database
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.