apache / apache/paimon

[Feature] Multi-Location Management

Open
#3,627 17 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

Currently, Paimon metadata does not store the absolute paths of files but uses relative paths to construct the absolute file paths instead. This is very cool because it saves a long string of identical path prefixes.
However, this also limits scenarios that require the use of different locations. For example, our warehouse is an internally hosted HDFS cluster, but for the purpose of saving resources, we would like to implement tiered storage. This means keeping only the hot data in the internal HDFS cluster and moving the cold data to public cloud object storage, which can save a lot of costs. But without support for table location management, we cannot achieve this in Paimon.

### Solution

Therefore, I suggest introducing **the ability to manage relative paths** in Paimon (this does not include the management of metadata paths such as snapshots and schemas, as these metadata always relies on the warehouse path), allowing table data to be stored in different locations.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how Paimon metadata currently constructs absolute data paths from relative paths, while keeping snapshot and schema metadata tied to the warehouse path. Review the issue's discussion for settled design constraints. Done means table data can use different locations, including cloud object storage, without relocating warehouse metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.