apache / apache/hudi

Support/Fix Byte data type w/ partition stats

Open
#17,367 1 comment 0 reactions 1 assignee Assigned to @nsivabalan View on GitHub
area:metadata-table from-jira priority:high type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

While working towards making partition stats default, we ran into an issue, with Byte data type 

[https://github.com/apache/hudi/pull/12671]

 

min max values when merging multiple values did not align w/ manually computed stats. Check for column "c7" in tests in TestColStatsIndex. 

 

To reproduce:

switch data type of C7 to "Byte". 

and run 

TestColumnStatsIndex.testMetadataColumnStatsIndex. 

 

comment out 

```

assertEquals(asJson(sort(expectedColStatsIndexTableDf, validationSortColumns)),
asJson(sort(transposedColStatsDF.drop("fileName"), validationSortColumns)))

```

in ColumnStatIndexTestBase.

 

Run the test for COW table. you may find the issue w/ below validation

```

assertEquals(asJson(sort(manualColStatsTableDF.drop(colsToDrop: _*), pValidationSortColumns)),
asJson(sort(pTransposedColStatsDF.drop(colsToDrop: _*), pValidationSortColumns)))

```

 

 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-8909
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9100
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.