apache / apache/hudi

Comprehensive mechanism around cleaning the archived timeline

Open
#14,959 0 comments 0 reactions 0 assignees View on GitHub
from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

At present, Hoodie's archive file grows indefinitely, which is more serious for dfs that does not support append.
After PR https://github.com/apache/hudi/pull/4078, now users will have some way to trim the archive files and not keep expanding indefinitely.
But as the document said *WARNING: do not use this config unless you know what you're doing. If enabled, details of older archived instants are deleted, resulting in information loss in the archived timeline, which may affect tools like CLI and repair. Only enable this if you hit severe performance issues for retrieving archived timeline.*

So we need a more comprehensive mechanism around cleaning the archived timeline.

(1) Rewrite the archived timeline content into a smaller number of files
(2) When deleting the archived files, make sure the table does not have any corresponding base or log files from the contained instants, so there is essentially no information loss of the table states.

As we know this two operation is pretty heavy, maybe we could build a new tool instead of a inner service to make it happen.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3038
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing PR 4078 and the linked HUDI-3038 JIRA issue to understand the existing archive-trimming behavior and its information-loss warning. Define how archive contents should be compacted and how deletion must account for base and log files from the contained instants; done means both operations are specified and validated without losing table-state information.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.