Check file listing for common partitions when partition list mismatches in the validation
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
{code:java}
org.apache.hudi.exception.HoodieValidationException: Compare Partitions Failed! AllPartitionPathsFromFS : [2021/12/1, 2022/1/24, 2022/1/25, 2022/1/26, 2022/1/27, 2022/1/28, 2022/1/29, 2022/1/30, 2022/1/31, 2022/2/1, 2022/2/2] and allPartitionPathsMeta : [2021/12/1, 2021/12/2, 2022/1/24, 2022/1/25, 2022/1/26, 2022/1/27, 2022/1/28, 2022/1/29, 2022/1/30, 2022/1/31, 2022/2/1, 2022/2/2] {code}
When the partition listing is different between FS and metadata table, we can still check the file listing for common partitions and print the result. For the partition that only appears in one side, we can check if it is empty or not. If it is, we can mark it as matching; otherwise, we throw an error for the partition.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3880
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-1292
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the validation path that raises HoodieValidationException and compare the filesystem and metadata partition lists described in the issue. Review JIRA HUDI-3880 and the HUDI-1292 epic for context; done means common partitions are checked, one-sided empty partitions are accepted, and nonempty mismatches still fail with a useful result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100