apache / apache/hudi

Performance degradation due to increased active timeline instants - Request for decoupling archival from savepoints (infinite time travel)

Open
#18,686 1 comment 1 reaction 0 assignees View on GitHub
type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Task Description

**What needs to be done:**
This issue is based on a discussion in the Hudi Slack channel regarding performance degradation with a large active timeline: https://apache-hudi.slack.com/archives/C4D716NPQ/p1774948526496729

Propose and/or implement a solution to decouple savepoints from timeline archival (e.g., enable archival without losing restore capability).

Additionally, update documentation to clearly state the impact of `hoodie.archive.beyond.savepoint` on savepoint restore behaviour.

**Why this task is needed:**

We are running a streaming pipeline writing to multiple Hudi MOR tables with:
- Async compaction and cleaner
- Commit frequency: every 5 minutes
- Savepoints retained for 7 days (1 per 24 hours)

Savepoints are required for our backup/recovery strategy and cannot be reduced. However, savepoints block archival of commits in the timeline, leading to continuous timeline growth and noticeable performance degradation in both reads and writes.

Currently, the config `hoodie.archive.beyond.savepoint` allows archival beyond savepoints, but at the cost of losing savepoint restore capability (i.e., savepoints become non-recoverable): -> https://github.com/apache/hudi/pull/6239

Hence, to resolve this, we need decoupling of savepoint from the timeline archival process, so that we can have "Restore capability" without having significant Performance degradation.

JFI: This task request was already part of this Jira: https://issues.apache.org/jira/browse/HUDI-4500. But since, Hudi is moved to Github Issues, I am creating this.

### Task Type

Performance optimization

### Related Issues

**Parent feature issue:** https://issues.apache.org/jira/browse/HUDI-4500
**Related issues:** https://issues.apache.org/jira/browse/HUDI-4501

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the behavior of hoodie.archive.beyond.savepoint and the linked PR 6239, then review the HUDI-4500 and HUDI-4501 requirements. The work is done when timeline archival no longer removes savepoint restore capability and the documentation clearly explains the configuration's restore impact.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.