[BUG] pyorc does not read string column statistics of cuDF generated files
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
When reading the statistics for an ORC file written by cuDF, the result for sum is wrong when read using cuDF and absent when using pyorc.
```python
In [1]: import cudf
In [2]: import pyorc
In [3]: gdf = cudf.DataFrame({'b':[1,7], 'a':['Badam khao', 'roz']})
In [4]: gdf.to_orc("temp.orc")
In [5]: cudf.io.orc.read_orc_statistics(["temp.orc"])
Out[5]:
([{'col0': {'number_of_values': 2},
'b': {'number_of_values': 2, 'minimum': 1, 'maximum': 7, 'sum': 8},
'a': {'number_of_values': 2,
'minimum': 'Badam khao',
'maximum': 'roz',
'sum': -7}}],
[{'col0': {'number_of_values': 2},
'b': {'number_of_values': 2, 'minimum': 1, 'maximum': 7, 'sum': 8},
'a': {'number_of_values': 2,
'minimum': 'Badam khao',
'maximum': 'roz',
'sum': -7}}])
In [6]: f = open("temp.orc", 'rb')
In [7]: r = pyorc.Reader(f)
In [8]: r[1].statistics
Out[8]:
{'has_null': False,
'number_of_values': 2,
'minimum': 1,
'maximum': 7,
'sum': 8,
'kind': }
In [9]: r[2].statistics
Out[9]: {'has_null': False, 'number_of_values': 2, 'kind': }
```
#### Expected result
Sum statistics contains the sum of lengths of all the strings in the column. We do correctly compute this in libcudf, so it should be present when reading with pyorc and correct when reading with cudf.
There's two issues here:
- [x] String sum statistics are encoded incorrectly (will be fixed by https://github.com/rapidsai/cudf/pull/11740)
- [ ] pyroc does not read cuDF-written ORC string statistics
Contributor guide
Assessment
This issue has not been assessed yet.