apache / apache/hudi

Re-evaluate handling of LogBlock appends when Compaction is pending

Open
#15,001 0 comments 0 reactions 0 assignees View on GitHub
area:table-service component:compaction from-jira priority:high type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently, when (async) Compaction for particular File Group has been scheduled but not yet completed, if writer will try to append additional Log Blocks to the same file-group following will occur:
# FileSystemView (when fetched), will check whether any compaction is pending and if it's it will inject "phantom" (ie non-existent) log-file into the existing FileSlice, which will have the same FileGroup name, but will bear instant of the scheduled Compaction commit (on the timeline) in its name (as opposed to the instant of the base-file)
# Writer will pick up such log-file as the latest
# Writer will write into such "phantom" log-file

[REF: https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java#L199|https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java#L199]

 

This posses following problems: 
* Reader now has to be aware of such handling and therefore always include pending compaction instants into its timeline when fetching the FileSystemView, as otherwise it will miss newly added log-files.
* This pushes the decision-making point of where writes should be channeled down into FileSystemView, which is clearly alien to its scope of responsibilities.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3302
- Type: Task

---

## Comments

30/Jan/22 17:59;shivnarayan;[~alexey.kudinkin] : may I know if there is any correctness issue or its about layering and abstractions. If correctness issue, wanted to target for 0.11. If not, can you tag w/ 0.12 (fix version)

 ;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java around line 199 and trace how pending compaction affects FileSlice selection and LogBlock appends. Then inspect the writer and reader paths involved. Done should remove the need for phantom log files or pending-compaction timeline workarounds while preserving correct appends and reads.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.