apache / apache/hudi

Support bootstrap files in file group reader-based compaction

Open
#17,320 2 comments 0 reactions 1 assignee Claimed by @yihua View on GitHub
from-jira priority:blocker status:pr-available type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

testMetadataBootstrapMORPartitionedInlineCompactionOn fails with file group reader-based compaction (HoodieSparkMergeHandleV2).  Currently if the compaction plan and operations contain bootstrap files, the compaction goes through the old flow using the regular HoodieMergeHandle.  We should support bootstrap files in file group reader-based compaction in HoodieSparkMergeHandleV2.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-8632
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9107
- Fix version(s):
- 1.1.0

---

## Comments

22/Jan/25 05:52;yihua;Found another issue: HUDI-8896.  Right now reading partition columns for bootstrapped table (bootstrap skeleton + data file) relies on engine-specific handling, not in the file group reader.  So compaction and clustering using the file group reader to read out records from metadata-only bootstrapped table can encounter nulls in the partition columns, causing the new base file written to have nulls in the partition columns.;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.