[BUG] Malformed fixed length byte array Parquet file loads corrupted data instead of error
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 274
Description
**Describe the bug**
Using libcudf to load a Parquet file that is malformed "succeeds" by producing a table with some corrupted rows rather than returning an error as expected. Spark 3.5, parquet-mr 1.13.1, and pyarrow 13 all produce unexpected EOF errors when trying to load the same file.
**Steps/Code to reproduce bug**
Load https://github.com/apache/parquet-testing/blob/master/data/fixed_length_byte_array.parquet using libcudf. Note that it will produce a table with 1000 rows with no nulls, and some of the rows have a list of bytes longer than 4 entries. According to the [docs for the file](https://github.com/apache/parquet-testing/blob/master/data/fixed_length_byte_array.md), the data is supposed to be a single column with a fixed-length byte array of size 4, yet some rows load with more than four bytes, some with no bytes.
**Expected behavior**
libcudf should return an error when trying to load the file rather than producing corrupted rows.
Contributor guide
Research direction
Start by loading the linked fixed_length_byte_array.parquet file with libcudf and compare its result with the unexpected EOF errors from Spark 3.5, parquet-mr 1.13.1, and pyarrow 13. Check the fixed-length byte array handling against the file documentation. Done means libcudf rejects this malformed file with an error instead of returning corrupted rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100