apache / apache/hudi

File slice instantiation for MDT file groups

Open
#17,241 1 comment 0 reactions 1 assignee Assigned to @nsivabalan View on GitHub
area:metadata-table from-jira priority:high type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

here is what a typical file group instantiation of MDT partition looks like

t10: create a dummy commit w/ base commit time "0000000". 

So this will create a log file w/ dummy delete block. 

Immediately following this, we take the bulk_insert which will create a new file slice but w/ same commit time. 

base_file_00000.parquet. 

Theoretically, these both belong to diff file slices and when latest snapshot is read, only latest base file should be read. but as of now, we consider the log file also as latest and read it. Since its dummy delete log block, there is no correctness issue here. 

 

Just some code clean up is required. 

 

this is an issue only w/ a fresh table. 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-7101
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9176
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.