On-demand API for backing up segments of offline table back to deepstore
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
We have an offline table for which the deepstore data got deleted. We want to copy the data back from Pinot-servers to deepstore. At present, there is no support for this in case of offline segments.
Will having an on-demand API which takes in the tableName(offline) and segmentName as params and backs up the segment data to the url mentioned in the segmentMetadata be helpful in this scenario?
I also thought of writing a script to download each segment and then upload it to deepstore via the script. The issue is with offline tables, it first checks if the url path mentioned in segmentMetadata exists or not and errors out. Should we either allow downloading segments for offline tables from pinot-server directly to make this working?
Contributor guide
Research direction
Start by reviewing how offline tables handle segment downloads and how the segmentMetadata URL is checked. Compare the proposed on-demand API with the script-based download and re-upload flow, then define the supported request inputs, backup behavior, and error handling before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100