[Feature] Introduce Archive ability
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
Introduce Archive ability to object store just like OSS and S3.
Spark SQL:
```sql
ALTER TABLE t PARTITION(dt = '20250101') ARCHIVE;
ALTER TABLE t PARTITION(dt = '20250101') COLD ARCHIVE;
ALTER TABLE t PARTITION(dt = '20250101') RESTORE ARCHIVE;
ALTER TABLE t PARTITION(dt = '20250101') UNARCHIVE;
```
For this SQL, we should initiate a task to perform distributed archiving, archiving all files in the partition for each file.
We can introduce methods to `FileIO`:
```java
Optional archive(Path path, StorageType type);
void restoreArchive(Path path, Duration duration);
Optional unarchive(Path path, StorageType type);
enum StorageType {
Standard("Standard"),
Archive("Archive"),
ColdArchive("ColdArchive"),
}
```
Archive and unarchive can return new path, some file systems do not support in place archiving and require generating additional paths, this may need to commit table too.
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the Spark SQL ALTER TABLE partition handling and the FileIO implementations named in the issue. Clarify how distributed partition tasks should archive, restore, and unarchive every file, including new paths and table commits. Done means the listed SQL commands and FileIO operations work across supported object stores with appropriate validation and coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark, sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100