apache / apache/arrow-rs

arrow-csv: no opt-in recovery for rows with more fields than the schema

Open
#10,578 1 comment 0 reactions 1 assignee Claimed by @srijithr03 View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

### Is your feature request related to a problem or challenge?

`ReaderBuilder::with_truncated_rows(true)` recovers a row that has fewer fields
than the schema by padding it. There is no equivalent for a row that has more
fields than the schema. That case always fails:

```
Csv error: incorrect number of fields for line 3, expected 3 got 4
```

The flag is documented as controlling truncated rows, so the behaviour is
consistent with its name. The gap is that a caller who wants to tolerate ragged
input has no way to express that for both directions. Most CSV readers treat
this as a single setting: the `csv` crate has `flexible`, which accepts records
of any length, and pandas has `on_bad_lines`.

Repro against arrow-csv 59.1.0, with the flag in both positions:

```rust
use arrow_csv::ReaderBuilder;
use arrow_schema::{DataType, Field, Schema};
use std::io::Cursor;
use std::sync::Arc;

fn main() {
let schema = Arc::new(Schema::new(vec![
Field::new("name", DataType::Utf8, true),
Field::new("age", DataType::Utf8, true),
Field::new("city", DataType::Utf8, true),
]));
let csv = "name,age,city\nAlice,25,Rome\nBob,30,Milan,EXTRA\n";

for allow in [true, false] {
let reader = ReaderBuilder::new(schema.clone())
.with_header(true)
.with_truncated_rows(allow)
.build(Cursor::new(csv))
.unwrap();
// Both settings error on line 3.
println!("with_truncated_rows({allow}): {:?}", reader.into_iter().next().unwrap().err());
}
}
```

### Describe the solution you'd like

An opt-in that discards fields past the schema width, so both directions of
raggedness can be recovered by configuration.

Naming is the open question, and I do not have a strong preference:

- `with_ragged_rows(bool)`, covering both directions in one flag, closest to the
`csv` crate's `flexible`.
- An enum such as `ExtraFields::{Error, Ignore}`, if the two directions are
worth controlling independently.

The second is more flexible and keeps `with_truncated_rows` meaning what it says
today. The first is simpler and matches what callers usually want.

### Describe alternatives you've considered

Pre-scanning the input with the `csv` crate to find the widest record, building
the Arrow schema at that width, then projecting the surplus columns away after
decoding. This is what we shipped. It works, but it costs a second full read of
the input and produces a schema that does not describe the data, purely so that
the decoder has somewhere to put fields that are going to be discarded.

Rejecting the file outright was the other option. We chose against it because
the same input parses without complaint through our non-Arrow engine, so the
choice of engine would have changed whether a file was readable.

### Additional context

Raised separately from #10577 at a maintainer's suggestion. The two are
independent: #10577 asks for a count of rows already being repaired, which is
additive and cheap, while this one changes what the decoder accepts and is the
larger change of the two.

Context for why both directions matter to us: dataprof is a data profiler, and
a row with a field count that disagrees with the header is a structural
violation we report on regardless of which direction it disagrees in. Our
Arrow-backed CSV engine was the only one that rejected one direction while
silently repairing the other
([dataprof#470](https://github.com/AndreaBozzo/dataprof/issues/470)).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.