apache / apache/arrow-rs

Arrow-csv does not update null flag in `Field` on inference

Open
#9,380 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Describe the bug**

The Format infer_schema does not update the null flag (by default it's true).

**To Reproduce**

Run the following on any csv file and the Fields will be always null even when they are not supposed to.
```
use std::fs::File;
use arrow::csv::Format;

let mut file = File::open(path)?;
let (schema, _) = Format::default()
.with_header(source.metadata.has_header)
.infer_schema(&mut file, Some(source.metadata.num_rows))?;
```

The line of code in infer_schema defaulting to true
```
// build schema from inference results
let fields: Fields = column_types
.iter()
.zip(&headers)
.map(|(inferred, field_name)| Field::new(field_name, inferred.get(), true))
.collect();
```

**Expected behaviour**

On running on a csv, the null flag should be updated accordingly if the field inferred is actually null or not. Besides, there is a disconnect to use the null regex and the null flag always being true.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start at the CSV `Format::infer_schema` entry point and inspect the schema-building code shown in the issue, especially how null inference and the null regex are used. Reproduce the behavior with the provided Rust snippet on a CSV containing nullable and non-nullable fields. Done means inferred `Field` null flags reflect the data and the existing null handling remains consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.