apache / apache/pinot

Tool for migrating from one deep store to another

Open
#7,275 4 comments 0 reactions 0 assignees View on GitHub
good first issue help wanted
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

A use case for this is, say someone started with controller local as deep store, and then wants to migrate to using Azure as deep store. If they enable this, the newer segments will use Azure as deep store, but the older segments will still remain on controller local disk.
The only way to move the old segments is to manually copy them over to Azure, and then modify the segment zk metadata.

An admin tool for migrating from deep store X to Y would be handy.

Contributor guide

Open the contributing guide

Research direction

Start by defining how the admin tool should migrate older segments from one deep store to another, using the controller-local-to-Azure example in the issue. Determine the required segment metadata update and consider how completion will verify that the old segments are available in the new deep store.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, java
Domain
databases, distributed-systems, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.