apache / apache/hudi

[SUPPORT] No way to clean `archived/` folder

Open
#10,930 7 comments 0 reactions 0 assignees View on GitHub
area:table-service type:community-support type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

There's no way to control `archived/` folder size and no way to trigger its cleaning.

We have a long running table which accumulated a lot of archives (~100 GB) which now damages cleaner performance and overall performance of ingestion process.

**To Reproduce**

Steps to reproduce the behavior:

1. Go to All Configuration in Hudi Site
2. Check for all settings that control archived/ folder of hudi
3. Ensure there is none

**Expected behavior**

There should be a description somewhere in documentation of stating how to upkeep `archived/` folder.

Upkeep of archived/ folder should be delegated to cleaner.

**Environment Description**

* Hudi version : 0.14.0

* Spark version : 3.4.1

* Hive version :

* Hadoop version :

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) :

**Additional context**

Related slack thread: https://apache-hudi.slack.com/archives/C4D716NPQ/p1711531654297129

**Stacktrace**

```Add the stacktrace of the error.```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the All Configuration page and the cleaner/archive behavior for Hudi 0.14.0 on Spark and S3. Compare the documented controls and triggering options with the reported 100 GB accumulation. Done should document archive upkeep and provide a clear, agreed way to control or trigger cleanup.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.