apache / apache/hudi

Revisit handling of pending compaction in file-system view

Open
#14,478 0 comments 0 reactions 0 assignees View on GitHub
area:core area:table-service component:compaction from-jira priority:medium type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

This came out of code review for Timeline Server. 

[https://github.com/apache/incubator-hudi/pull/600/files#r276825586]

We need to investigate if pending compaction operation can be done in a better way so that it can easily be applied to File Stiching as well.

 

The current approach stems from the requirement that we have to satisfy both use-case : 

 1. Fetch latest file slice without any regard for pending compaction for appending

 2. Fetch Merged file slices for realtime views with pending compaction factored

To avoid duplicate storage, we are doing lazy merging due to pending compaction. 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-103
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the Timeline Server context and the code-review discussion in PR #600, then trace how pending compaction is handled for the file-system view. Compare the two stated use cases: latest file slices for appending and merged slices for realtime views. Done means defining and implementing a better approach that can also be applied to File Stitching without duplicate storage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.