NVIDIA / NVIDIA/cudf

[BUG] pyorc does not read string column statistics of cuDF generated files

Open
#9,313 4 comments 0 reactions 0 assignees View on GitHub
bug cuIO libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

When reading the statistics for an ORC file written by cuDF, the result for sum is wrong when read using cuDF and absent when using pyorc.

```python
In [1]: import cudf

In [2]: import pyorc

In [3]: gdf = cudf.DataFrame({'b':[1,7], 'a':['Badam khao', 'roz']})

In [4]: gdf.to_orc("temp.orc")

In [5]: cudf.io.orc.read_orc_statistics(["temp.orc"])
Out[5]:
([{'col0': {'number_of_values': 2},
'b': {'number_of_values': 2, 'minimum': 1, 'maximum': 7, 'sum': 8},
'a': {'number_of_values': 2,
'minimum': 'Badam khao',
'maximum': 'roz',
'sum': -7}}],
[{'col0': {'number_of_values': 2},
'b': {'number_of_values': 2, 'minimum': 1, 'maximum': 7, 'sum': 8},
'a': {'number_of_values': 2,
'minimum': 'Badam khao',
'maximum': 'roz',
'sum': -7}}])

In [6]: f = open("temp.orc", 'rb')

In [7]: r = pyorc.Reader(f)

In [8]: r[1].statistics
Out[8]:
{'has_null': False,
'number_of_values': 2,
'minimum': 1,
'maximum': 7,
'sum': 8,
'kind': }

In [9]: r[2].statistics
Out[9]: {'has_null': False, 'number_of_values': 2, 'kind': }
```

#### Expected result
Sum statistics contains the sum of lengths of all the strings in the column. We do correctly compute this in libcudf, so it should be present when reading with pyorc and correct when reading with cudf.

There's two issues here:

- [x] String sum statistics are encoded incorrectly (will be fixed by https://github.com/rapidsai/cudf/pull/11740)
- [ ] pyroc does not read cuDF-written ORC string statistics

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.