apache / apache/datafusion

Empty strings in CSV files aren't being interpreted as null when using a `Dictionary(_, Utf8)`

Open
#12,041 8 comments 0 reactions 1 assignee Claimed by @edmondop View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Related to #7797

Empty strings in CSV files aren't being interpreted as null when using a `Dictionary(_, Utf8)`

### To Reproduce

Create a simple `input.csv` file like this:

```
id,name
1,
2,bob
```

Run the following code:

```rust
#[tokio::main]
async fn main() -> Result<(), DataFusionError> {
let ctx = SessionContext::new();

let format = CsvFormat::default();
let listing_options = ListingOptions::new(Arc::new(format));
ctx.register_listing_table(
"input",
"input.csv",
listing_options.clone(),
Some(Arc::new(Schema::new(vec![
Field::new("id", DataType::Utf8, false),
Field::new(
"name",
DataType::Dictionary(Box::new(DataType::UInt8), Box::new(DataType::Utf8)),
true,
),
]))),
None,
)
.await?;

let results = ctx
.table("input")
.await?
.filter(col("name").is_not_null())?
.collect()
.await?;

let pretty_results = arrow::util::pretty::pretty_format_batches(&results)?.to_string();

println!("{}", pretty_results);

Ok(())
}
```

### Expected behavior

I was expecting the output to look like this:

```
+----+------+
| id | name |
+----+------+
| 2 | bob |
+----+------+
```

But the full dataset is returned instead:

```
+----+------+
| id | name |
+----+------+
| 1 | |
| 2 | bob |
+----+------+
```

### Additional context

Tested on v41.0.0

Replace `DataType::Dictionary(Box::new(DataType::UInt8), Box::new(DataType::Utf8))` with `DataType::Utf8` and it works.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.