[BUG] Bools written by cuIO ORC writer don't match when read by pyarrow/pyorc
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
When writing a large dataframe with bool column using cuIO ORC writer, the result of reading the file back using pyarrow does not match the input dataframe. However when reading back from cudf's ORC reader it matches.
```python
import pandas as pd
import numpy as np
import pyarrow as pa
import pyorc
import cudf
np.random.seed(0)
from cudf._lib.null_mask import bitmask_allocation_size_bytes
def random_bitmask(size):
sz = bitmask_allocation_size_bytes(size)
data = np.random.randint(0, 255, dtype="u1", size=sz)
return data.view("i1")
size = 6000000
arr = np.random.randint(low=0, high=2, size=size).astype(np.bool)
s = cudf.Series.from_masked_array(arr, random_bitmask(size))
gdf = cudf.DataFrame({"col_bool": s})
# write with cuIO
fname = "brokenbool.orc"
gdf.to_orc(fname)
# read with pyarrow
pdf = pa.orc.ORCFile(fname).read().to_pandas()
# the sum doesn't match
print(gdf.col_bool.sum(), pdf.col_bool.sum())
# read with pyorc
file = open(fname, 'rb')
data = pyorc.Reader(file).read()
pdf = pd.DataFrame(data, columns=["col_bool"])
# sum matches pyarrow but not original df
print(gdf.col_bool.sum(), pdf.col_bool.sum())
# reading with cuIO gives the correct result
print(gdf.col_bool.sum(), cudf.read_orc(fname).col_bool.sum())
```
Note that this doesn't occur when there are no nulls in the input.
Contributor guide
Assessment
This issue has not been assessed yet.