apache / apache/arrow-rs

Add fuzz regression testing to parquet/arrow/csv readers

Open
#9,358 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The arrow-testing repository has several data files that caused issues with the C/C++ implementation
* https://github.com/apache/arrow-testing/tree/master/data/arrow-ipc-stream
* https://github.com/apache/arrow-testing/tree/master/data/arrow-ipc-file
* https://github.com/apache/arrow-testing/tree/master/data/parquet/fuzzing
* https://github.com/apache/arrow-testing/tree/master/data/csv/fuzzing

It would be nice to add tests in this repository that ensure the parquet/arrow/csv readers behave "nicely" when reading such files.

The definition of "nice" would be "no panics, and errors when appropriate"

**Describe the solution you'd like**
1. Add tests to try and read all the above mentioned invalid files
2. If any cause panics, temporarily skip them in the tests and file a ticket to track fixing the panics

**Describe alternatives you've considered**

**Additional context**
This was inspired while reviewing this doc from @pitrou on arrow security guidelines:
- https://github.com/apache/arrow/pull/48870

Contributor guide

Open the contributing guide

Research direction

Review the arrow-testing paths data/arrow-ipc-stream, data/arrow-ipc-file, data/parquet/fuzzing, and data/csv/fuzzing, then locate the repository's parquet, Arrow, and CSV reader test entry points. Exercise every listed invalid file and verify that readers do not panic and return errors when appropriate; temporarily skip panic cases and file tracking tickets for them.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.