NVIDIA / NVIDIA/cudf

[BUG] Bools written by cuIO ORC writer don't match when read by pyarrow/pyorc

Open
#6,763 8 comments 0 reactions 0 assignees View on GitHub
bug cuIO libcudf Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

When writing a large dataframe with bool column using cuIO ORC writer, the result of reading the file back using pyarrow does not match the input dataframe. However when reading back from cudf's ORC reader it matches.

```python
import pandas as pd
import numpy as np
import pyarrow as pa
import pyorc
import cudf

np.random.seed(0)
from cudf._lib.null_mask import bitmask_allocation_size_bytes

def random_bitmask(size):
sz = bitmask_allocation_size_bytes(size)
data = np.random.randint(0, 255, dtype="u1", size=sz)
return data.view("i1")

size = 6000000
arr = np.random.randint(low=0, high=2, size=size).astype(np.bool)
s = cudf.Series.from_masked_array(arr, random_bitmask(size))
gdf = cudf.DataFrame({"col_bool": s})

# write with cuIO
fname = "brokenbool.orc"
gdf.to_orc(fname)

# read with pyarrow
pdf = pa.orc.ORCFile(fname).read().to_pandas()

# the sum doesn't match
print(gdf.col_bool.sum(), pdf.col_bool.sum())

# read with pyorc
file = open(fname, 'rb')
data = pyorc.Reader(file).read()
pdf = pd.DataFrame(data, columns=["col_bool"])

# sum matches pyarrow but not original df
print(gdf.col_bool.sum(), pdf.col_bool.sum())

# reading with cuIO gives the correct result
print(gdf.col_bool.sum(), cudf.read_orc(fname).col_bool.sum())
```

Note that this doesn't occur when there are no nulls in the input.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.