NVIDIA / NVIDIA/cudf

[BUG] String columns written with `fastparquet` seem to be read incorrectly via CUDF's Parquet reader

Open
#14,258 6 comments 0 reactions 0 assignees View on GitHub
0 - Backlog bug cuIO libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Description**

This was uncovered in [Spark tests](https://github.com/NVIDIA/spark-rapids/pull/9366) that compare Parquet read/write compatibility with [`fastparquet`](https://fastparquet.readthedocs.io/en/latest/index.html).

The last row of a String column written with `fastparquet` seems to be interpreted by CUDF as having more null characters at the end than expected.

**Repro**

I'll spare the Scala/Spark details in this bug. [Here](https://github.com/NVIDIA/spark-rapids/files/12812419/fastparquet_string.zip) is a zipped Parquet file that seems to be read differently in CUDF.

From https://github.com/NVIDIA/spark-rapids/issues/9387:
```
GPU COLUMN LENGTH - NC: 0 DATA: DeviceMemoryBufferView{address=0x30a003400, length=20, id=-1} VAL: DeviceMemoryBufferView{address=0x30a001e00, length=64, id=-1}
COLUMN LENGTH - STRING
0 "all" 616c6c
1 "the" 746865
2 "leaves" 6c65617665730000000000000000
```

It would be good to check with the CUDF native Parquet reader, and compare against the results from `parquet-mr`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.