Support/Fix Byte data type w/ partition stats
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
While working towards making partition stats default, we ran into an issue, with Byte data type
[https://github.com/apache/hudi/pull/12671]
min max values when merging multiple values did not align w/ manually computed stats. Check for column "c7" in tests in TestColStatsIndex.
To reproduce:
switch data type of C7 to "Byte".
and run
TestColumnStatsIndex.testMetadataColumnStatsIndex.
comment out
```
assertEquals(asJson(sort(expectedColStatsIndexTableDf, validationSortColumns)),
asJson(sort(transposedColStatsDF.drop("fileName"), validationSortColumns)))
```
in ColumnStatIndexTestBase.
Run the test for COW table. you may find the issue w/ below validation
```
assertEquals(asJson(sort(manualColStatsTableDF.drop(colsToDrop: _*), pValidationSortColumns)),
asJson(sort(pTransposedColStatsDF.drop(colsToDrop: _*), pValidationSortColumns)))
```
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-8909
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9100
- Fix version(s):
- 1.1.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.