Adding support to fetch ECTR from lsm timeline
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Describe the problem you faced
For datasets having very infrequent cleans, ECTR could go into archived timeline. with 1.x, we do have lsm timeline and hence should be able to add support to fetch the ECTR from lsm timeline if we can't find them in active timeline.
Ref patch that we added for 0.x to get around this:
https://github.com/apache/hudi/pull/18380
### To Reproduce
1.
2.
3.
4.
### Expected behavior
If ECTR falls behind active timeilne, clean planner should seamlessly fetch it from lsm timeline and proceed w/ planning instead of doing full table planning.
### Environment Description
* Hudi version:
* Spark version:
* Flink version:
* Hive version:
* Hadoop version:
* Storage (HDFS/S3/GCS..):
* Running on Docker? (yes/no):
### Additional context
_No response_
### Stacktrace
```shell
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the referenced 0.x patch in PR 18380 and the clean planner flow that looks for ECTR in the active timeline. Trace how the 1.x LSM timeline is accessed, then verify that planning retrieves ECTR from it when the active timeline has no entry and avoids full-table planning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100