apache / apache/hudi

Fix reading partition path field on metadata bootstrap table

Open
#17,358 1 comment 0 reactions 1 assignee Claimed by @yihua View on GitHub
from-jira priority:blocker type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

When adding strict data validation within testMetadataBootstrapMORPartitionedInlineCompactionOn, the validation reveals that the partition path field reading fails (returns null) for some update records. 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-8837
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9108
- Fix version(s):
- 1.1.0

---

## Comments

10/Jan/25 00:12;yihua;The test is added in https://github.com/apache/hudi/pull/12490. Right now the validation excludes partition column.  When adding that in the validation, the validation fails.

 
{code:java}
def assertDfEquals(df1: DataFrame, df2: DataFrame): Unit = {
    assertEquals(df1.count, df2.count)
    // TODO(HUDI-8723): fix reading partition path field on metadata bootstrap table
    assertEquals(0, df1.drop(partitionColName).except(df2.drop(partitionColName)).count)
    assertEquals(0, df2.drop(partitionColName).except(df1.drop(partitionColName)).count)
  } {code}
 

 ;;;

---

10/Jan/25 00:26;daviszhang;so we can remove the .drop(partitionColName) in the validation func you mentioned, ran all tests in the test suite, all green. Assigned back to you;;;

---

28/Jan/25 01:05;yihua;This is still an issue for reading the partition column value out from a bootstrapped file slice (merging skeleton and data files), using the file group reader only. Deferring this ticket to 1.0.2 release.;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.