apache / apache/hudi

Adding support to fetch ECTR from lsm timeline

Open
#18,536 0 comments 0 reactions 0 assignees View on GitHub
type:community-support
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Describe the problem you faced

For datasets having very infrequent cleans, ECTR could go into archived timeline. with 1.x, we do have lsm timeline and hence should be able to add support to fetch the ECTR from lsm timeline if we can't find them in active timeline.

Ref patch that we added for 0.x to get around this:
https://github.com/apache/hudi/pull/18380

### To Reproduce

1.
2.
3.
4.

### Expected behavior

If ECTR falls behind active timeilne, clean planner should seamlessly fetch it from lsm timeline and proceed w/ planning instead of doing full table planning.

### Environment Description

* Hudi version:
* Spark version:
* Flink version:
* Hive version:
* Hadoop version:
* Storage (HDFS/S3/GCS..):
* Running on Docker? (yes/no):

### Additional context

_No response_

### Stacktrace

```shell

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the referenced 0.x patch in PR 18380 and the clean planner flow that looks for ECTR in the active timeline. Trace how the 1.x LSM timeline is accessed, then verify that planning retrieves ECTR from it when the active timeline has no entry and avoids full-table planning.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.