apache / apache/hudi

Check file listing for common partitions when partition list mismatches in the validation

Open
#15,126 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

{code:java}
org.apache.hudi.exception.HoodieValidationException: Compare Partitions Failed! AllPartitionPathsFromFS : [2021/12/1, 2022/1/24, 2022/1/25, 2022/1/26, 2022/1/27, 2022/1/28, 2022/1/29, 2022/1/30, 2022/1/31, 2022/2/1, 2022/2/2] and allPartitionPathsMeta : [2021/12/1, 2021/12/2, 2022/1/24, 2022/1/25, 2022/1/26, 2022/1/27, 2022/1/28, 2022/1/29, 2022/1/30, 2022/1/31, 2022/2/1, 2022/2/2] {code}
When the partition listing is different between FS and metadata table, we can still check the file listing for common partitions and print the result.  For the partition that only appears in one side, we can check if it is empty or not.  If it is, we can mark it as matching; otherwise, we throw an error for the partition.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3880
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-1292

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the validation path that raises HoodieValidationException and compare the filesystem and metadata partition lists described in the issue. Review JIRA HUDI-3880 and the HUDI-1292 epic for context; done means common partitions are checked, one-sided empty partitions are accepted, and nonempty mismatches still fail with a useful result.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.