apache / apache/hudi

Fix HoodieTestTable harness to also properly validate Column Stats

Open
#15,103 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently `HoodieTestTable` generates invalid tables which prevent effective testing of the Column Stats Index:
# It does write empty Parquet files, making impossible to construct proper ColStats records for these
# It does set phony Commit Metadata omitting Schema (for aforementioned reasons), which breaks ColStats when trying to update records during Rollbacks/Cleanings as CS relies on TableSchemaResolver, which is not able to resolve schema for an invalid table

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3783
- Type: Bug
- Epic: https://issues.apache.org/jira/browse/HUDI-1822

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the HoodieTestTable harness and trace how it creates Parquet files and commit metadata. Check the Column Stats Index paths that use TableSchemaResolver during rollbacks and cleanings. Done means the harness produces valid tables with usable column-stat records and schema-bearing commit metadata, allowing those validation and update paths to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.