apache / apache/hudi

Enhance Metadata tests to validate col stats

Open
#15,951 0 comments 0 reactions 0 assignees View on GitHub
area:metadata-table from-jira priority:blocker priority:critical type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 4h
Merged PRs (30d)
112

Description

I happened to check for col stats tests in TestHoodieBackedMetadata and did not find any good tests that validates col stats records from MDT and bloom partition records. I do see some validations for log files, but none for base files. we do have one test named testColStatsPrefixLookup which tests col stats. but if we can add a separate validateColStats() method we can call it with validateMetadata() for any tests we have to gain more coverage. 

 

We do a sparate test class TestColumnStatsIndex for testing col stats. But would be good to enhance the validations in TestHoodieBackedMetadata tests.

we should enhance them for sure. 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-6206
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with TestHoodieBackedMetadata and its existing testColStatsPrefixLookup and log-file validations. Compare those checks with TestColumnStatsIndex, then define what validateColStats() should cover when called from validateMetadata(), including metadata-table and bloom-partition records and base files; done means the relevant column-stat records are validated in the existing tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.