Support Continue on Error in CSV Reader
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Describe the bug**
Attempting to read malformed CSV records where the number of columns doesn't match the schema leads to an out-of-memory error.
**To Reproduce**
See minimal example Rust code below.
[arrow_bug.txt](https://github.com/apache/arrow-rs/files/12575141/arrow_bug.txt)
**Expected behavior**
The CSV reader should start re-start reading from the first newline/delimiter after the malformed row is encountered.
Contributor guide
Research direction
Start with the minimal Rust reproduction in arrow_bug.txt and trace the CSV reader while processing a row whose column count does not match the schema. Confirm the malformed row can be skipped and reading resumes at the first newline or delimiter without an out-of-memory error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100