apache / apache/paimon

[Feature] Introduce Archive ability

Open
#5,510 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

Introduce Archive ability to object store just like OSS and S3.
Spark SQL:
```sql
ALTER TABLE t PARTITION(dt = '20250101') ARCHIVE;
ALTER TABLE t PARTITION(dt = '20250101') COLD ARCHIVE;

ALTER TABLE t PARTITION(dt = '20250101') RESTORE ARCHIVE;
ALTER TABLE t PARTITION(dt = '20250101') UNARCHIVE;
```
For this SQL, we should initiate a task to perform distributed archiving, archiving all files in the partition for each file.

We can introduce methods to `FileIO`:
```java
Optional archive(Path path, StorageType type);

void restoreArchive(Path path, Duration duration);

Optional unarchive(Path path, StorageType type);

enum StorageType {
Standard("Standard"),
Archive("Archive"),
ColdArchive("ColdArchive"),
}
```

Archive and unarchive can return new path, some file systems do not support in place archiving and require generating additional paths, this may need to commit table too.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the Spark SQL ALTER TABLE partition handling and the FileIO implementations named in the issue. Clarify how distributed partition tasks should archive, restore, and unarchive every file, including new paths and table commits. Done means the listed SQL commands and FileIO operations work across supported object stores with appropriate validation and coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.