Fix HoodieTestTable harness to also properly validate Column Stats
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently `HoodieTestTable` generates invalid tables which prevent effective testing of the Column Stats Index:
# It does write empty Parquet files, making impossible to construct proper ColStats records for these
# It does set phony Commit Metadata omitting Schema (for aforementioned reasons), which breaks ColStats when trying to update records during Rollbacks/Cleanings as CS relies on TableSchemaResolver, which is not able to resolve schema for an invalid table
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3783
- Type: Bug
- Epic: https://issues.apache.org/jira/browse/HUDI-1822
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the HoodieTestTable harness and trace how it creates Parquet files and commit metadata. Check the Column Stats Index paths that use TableSchemaResolver during rollbacks and cleanings. Done means the harness produces valid tables with usable column-stat records and schema-bearing commit metadata, allowing those validation and update paths to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100