NVIDIA / NVIDIA/cudf

[BUG] Malformed fixed length byte array Parquet file loads corrupted data instead of error

Open
#14,104 5 comments 0 reactions 0 assignees View on GitHub
1 - On Deck bug cuIO libcudf Spark
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
2d 22h
Merged PRs (30d)
274

Description

**Describe the bug**
Using libcudf to load a Parquet file that is malformed "succeeds" by producing a table with some corrupted rows rather than returning an error as expected. Spark 3.5, parquet-mr 1.13.1, and pyarrow 13 all produce unexpected EOF errors when trying to load the same file.

**Steps/Code to reproduce bug**
Load https://github.com/apache/parquet-testing/blob/master/data/fixed_length_byte_array.parquet using libcudf. Note that it will produce a table with 1000 rows with no nulls, and some of the rows have a list of bytes longer than 4 entries. According to the [docs for the file](https://github.com/apache/parquet-testing/blob/master/data/fixed_length_byte_array.md), the data is supposed to be a single column with a fixed-length byte array of size 4, yet some rows load with more than four bytes, some with no bytes.

**Expected behavior**
libcudf should return an error when trying to load the file rather than producing corrupted rows.

Contributor guide

Open the contributing guide

Research direction

Start by loading the linked fixed_length_byte_array.parquet file with libcudf and compare its result with the unexpected EOF errors from Spark 3.5, parquet-mr 1.13.1, and pyarrow 13. Check the fixed-length byte array handling against the file documentation. Done means libcudf rejects this malformed file with an error instead of returning corrupted rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.