[SUPPORT] No way to clean `archived/` folder
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
There's no way to control `archived/` folder size and no way to trigger its cleaning.
We have a long running table which accumulated a lot of archives (~100 GB) which now damages cleaner performance and overall performance of ingestion process.
**To Reproduce**
Steps to reproduce the behavior:
1. Go to All Configuration in Hudi Site
2. Check for all settings that control archived/ folder of hudi
3. Ensure there is none
**Expected behavior**
There should be a description somewhere in documentation of stating how to upkeep `archived/` folder.
Upkeep of archived/ folder should be delegated to cleaner.
**Environment Description**
* Hudi version : 0.14.0
* Spark version : 3.4.1
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) :
**Additional context**
Related slack thread: https://apache-hudi.slack.com/archives/C4D716NPQ/p1711531654297129
**Stacktrace**
```Add the stacktrace of the error.```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the All Configuration page and the cleaner/archive behavior for Hudi 0.14.0 on Spark and S3. Compare the documented controls and triggering options with the reported 100 GB accumulation. Done should document archive upkeep and provide a clear, agreed way to control or trigger cleanup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100