[BUG] The hasNull in ORC Statistics is incorrect
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
The ORC Statistics show hasNull is true but the column values are all non-nulls.
**Steps/Code to reproduce bug**
Refer to the reproduce in [13793](https://github.com/rapidsai/cudf/issues/13793), RAPIDS Accelerator bug is:[8826](https://github.com/NVIDIA/spark-rapids/issues/8826)
```
$ orc-tool meta TestAllNulls.orc
Processing data file TestAllNulls.orc [length: 267]
Structure for TestAllNulls.orc
File Version: 0.12 with ORIGINAL by ORC Java
Rows: 3
Compression: SNAPPY
Compression size: 262144
Calendar: Julian/Gregorian
Type: struct
Stripe Statistics:
Stripe 1:
Column 0: count: 3 hasNull: true
Column 1: count: 3 hasNull: true min: 1 max: 3 sum: 3
Column 2: count: 0 hasNull: true
File Statistics:
Column 0: count: 3 hasNull: true
Column 1: count: 3 hasNull: true min: 1 max: 3 sum: 3
Column 2: count: 0 hasNull: true
Stripes:
Stripe: offset: 3 data: 19 rows: 3 tail: 71 index: 50
Stream: column 0 section ROW_INDEX start: 3 length 7
Stream: column 1 section ROW_INDEX start: 10 length 26
Stream: column 2 section ROW_INDEX start: 36 length 17
Stream: column 1 section DATA start: 53 length 6
Stream: column 1 section LENGTH start: 59 length 5
Stream: column 2 section PRESENT start: 64 length 5
Stream: column 2 section DATA start: 69 length 3
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT
File length: 267 bytes
Padding length: 0 bytes
Padding ratio: 0%
```
This static is wrong:
Column 1: count: 3 hasNull: true min: 1 max: 3 sum: 3
This column values are 1,2,3. The hasNull should be false.
**Expected behavior**
Stripe Statistics and File Statistics report correct hasNull info.
**Additional context**
I'm checking the `count` statistics for nested types, seems also has problem. I'll report another issue after one more check.
Contributor guide
Assessment
This issue has not been assessed yet.