Arrow-csv does not update null flag in `Field` on inference
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Describe the bug**
The Format infer_schema does not update the null flag (by default it's true).
**To Reproduce**
Run the following on any csv file and the Fields will be always null even when they are not supposed to.
```
use std::fs::File;
use arrow::csv::Format;
let mut file = File::open(path)?;
let (schema, _) = Format::default()
.with_header(source.metadata.has_header)
.infer_schema(&mut file, Some(source.metadata.num_rows))?;
```
The line of code in infer_schema defaulting to true
```
// build schema from inference results
let fields: Fields = column_types
.iter()
.zip(&headers)
.map(|(inferred, field_name)| Field::new(field_name, inferred.get(), true))
.collect();
```
**Expected behaviour**
On running on a csv, the null flag should be updated accordingly if the field inferred is actually null or not. Besides, there is a disconnect to use the null regex and the null flag always being true.
**Additional context**
Contributor guide
Research direction
Start at the CSV `Format::infer_schema` entry point and inspect the schema-building code shown in the issue, especially how null inference and the null regex are used. Reproduce the behavior with the provided Rust snippet on a CSV containing nullable and non-nullable fields. Done means inferred `Field` null flags reflect the data and the existing null handling remains consistent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100