NVIDIA / NVIDIA/cudf

[BUG] The hasNull in ORC Statistics is incorrect

Open
#13,817 0 comments 0 reactions 0 assignees View on GitHub
2 - In Progress bug cuIO libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
The ORC Statistics show hasNull is true but the column values are all non-nulls.

**Steps/Code to reproduce bug**
Refer to the reproduce in [13793](https://github.com/rapidsai/cudf/issues/13793), RAPIDS Accelerator bug is:[8826](https://github.com/NVIDIA/spark-rapids/issues/8826)

```
$ orc-tool meta TestAllNulls.orc
Processing data file TestAllNulls.orc [length: 267]
Structure for TestAllNulls.orc
File Version: 0.12 with ORIGINAL by ORC Java
Rows: 3
Compression: SNAPPY
Compression size: 262144
Calendar: Julian/Gregorian
Type: struct

Stripe Statistics:
Stripe 1:
Column 0: count: 3 hasNull: true
Column 1: count: 3 hasNull: true min: 1 max: 3 sum: 3
Column 2: count: 0 hasNull: true

File Statistics:
Column 0: count: 3 hasNull: true
Column 1: count: 3 hasNull: true min: 1 max: 3 sum: 3
Column 2: count: 0 hasNull: true

Stripes:
Stripe: offset: 3 data: 19 rows: 3 tail: 71 index: 50
Stream: column 0 section ROW_INDEX start: 3 length 7
Stream: column 1 section ROW_INDEX start: 10 length 26
Stream: column 2 section ROW_INDEX start: 36 length 17
Stream: column 1 section DATA start: 53 length 6
Stream: column 1 section LENGTH start: 59 length 5
Stream: column 2 section PRESENT start: 64 length 5
Stream: column 2 section DATA start: 69 length 3
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT

File length: 267 bytes
Padding length: 0 bytes
Padding ratio: 0%
```

This static is wrong:
Column 1: count: 3 hasNull: true min: 1 max: 3 sum: 3
This column values are 1,2,3. The hasNull should be false.

**Expected behavior**
Stripe Statistics and File Statistics report correct hasNull info.

**Additional context**
I'm checking the `count` statistics for nested types, seems also has problem. I'll report another issue after one more check.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.