Revisit handling of pending compaction in file-system view
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
This came out of code review for Timeline Server.
[https://github.com/apache/incubator-hudi/pull/600/files#r276825586]
We need to investigate if pending compaction operation can be done in a better way so that it can easily be applied to File Stiching as well.
The current approach stems from the requirement that we have to satisfy both use-case :
1. Fetch latest file slice without any regard for pending compaction for appending
2. Fetch Merged file slices for realtime views with pending compaction factored
To avoid duplicate storage, we are doing lazy merging due to pending compaction.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-103
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the Timeline Server context and the code-review discussion in PR #600, then trace how pending compaction is handled for the file-system view. Compare the two stated use cases: latest file slices for appending and merged slices for realtime views. Done means defining and implementing a better approach that can also be applied to File Stitching without duplicate storage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100