apache / apache/pinot

On-demand API for backing up segments of offline table back to deepstore

Open
#10,353 1 comment 0 reactions 0 assignees View on GitHub
rest-api
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

We have an offline table for which the deepstore data got deleted. We want to copy the data back from Pinot-servers to deepstore. At present, there is no support for this in case of offline segments.
Will having an on-demand API which takes in the tableName(offline) and segmentName as params and backs up the segment data to the url mentioned in the segmentMetadata be helpful in this scenario?

I also thought of writing a script to download each segment and then upload it to deepstore via the script. The issue is with offline tables, it first checks if the url path mentioned in segmentMetadata exists or not and errors out. Should we either allow downloading segments for offline tables from pinot-server directly to make this working?

Contributor guide

Open the contributing guide

Research direction

Start by reviewing how offline tables handle segment downloads and how the segmentMetadata URL is checked. Compare the proposed on-demand API with the script-based download and re-upload flow, then define the supported request inputs, backup behavior, and error handling before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.