[Feature] Multi-Location Management
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
Currently, Paimon metadata does not store the absolute paths of files but uses relative paths to construct the absolute file paths instead. This is very cool because it saves a long string of identical path prefixes.
However, this also limits scenarios that require the use of different locations. For example, our warehouse is an internally hosted HDFS cluster, but for the purpose of saving resources, we would like to implement tiered storage. This means keeping only the hot data in the internal HDFS cluster and moving the cold data to public cloud object storage, which can save a lot of costs. But without support for table location management, we cannot achieve this in Paimon.
### Solution
Therefore, I suggest introducing **the ability to manage relative paths** in Paimon (this does not include the management of metadata paths such as snapshots and schemas, as these metadata always relies on the warehouse path), allowing table data to be stored in different locations.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how Paimon metadata currently constructs absolute data paths from relative paths, while keeping snapshot and schema metadata tied to the warehouse path. Review the issue's discussion for settled design constraints. Done means table data can use different locations, including cloud object storage, without relocating warehouse metadata.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100