Enhance Metadata tests to validate col stats
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 112
Description
I happened to check for col stats tests in TestHoodieBackedMetadata and did not find any good tests that validates col stats records from MDT and bloom partition records. I do see some validations for log files, but none for base files. we do have one test named testColStatsPrefixLookup which tests col stats. but if we can add a separate validateColStats() method we can call it with validateMetadata() for any tests we have to gain more coverage.
We do a sparate test class TestColumnStatsIndex for testing col stats. But would be good to enhance the validations in TestHoodieBackedMetadata tests.
we should enhance them for sure.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6206
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with TestHoodieBackedMetadata and its existing testColStatsPrefixLookup and log-file validations. Compare those checks with TestColumnStatsIndex, then define what validateColStats() should cover when called from validateMetadata(), including metadata-table and bloom-partition records and base files; done means the relevant column-stat records are validated in the existing tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, testing
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100