Unable to upload existing segments to a table
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
Hello,
I try to upload some old segments produced by an REALTIME table to an OFFLINE table.
It is working as expected using the API as explained here: https://docs.pinot.apache.org/build-with-pinot/ingestion/batch-ingestion/segment-upload#uri-push
But the API is for 1 segment and I want to upload an entire "directory" with many segments. For this a job using Pinot Admin seems to be the solution.
I tried using "SegmentUriPush" but it does not work because my segment are named "myTable__0__0__20260601T1557Z" with no tar.gz extension.
But this extension seems to be required when the job is used (this is not the case using the API).
Am I doing something wrong or is it a limitation that can be fixed in a future version ?
I use Azure Data Lake Storage as a deep store.
Thanks for your help.
Contributor guide
Research direction
Start with the SegmentUriPush job and compare its handling of segment names with the documented API URI-push behavior, especially for Azure Data Lake Storage. Confirm whether extensionless segments from the named directory are expected to upload; done means the job handles these segments consistently or the limitation is clearly established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, java
- Domain
- database
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100