Empty strings in CSV files aren't being interpreted as null when using a `Dictionary(_, Utf8)`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
Related to #7797
Empty strings in CSV files aren't being interpreted as null when using a `Dictionary(_, Utf8)`
### To Reproduce
Create a simple `input.csv` file like this:
```
id,name
1,
2,bob
```
Run the following code:
```rust
#[tokio::main]
async fn main() -> Result<(), DataFusionError> {
let ctx = SessionContext::new();
let format = CsvFormat::default();
let listing_options = ListingOptions::new(Arc::new(format));
ctx.register_listing_table(
"input",
"input.csv",
listing_options.clone(),
Some(Arc::new(Schema::new(vec![
Field::new("id", DataType::Utf8, false),
Field::new(
"name",
DataType::Dictionary(Box::new(DataType::UInt8), Box::new(DataType::Utf8)),
true,
),
]))),
None,
)
.await?;
let results = ctx
.table("input")
.await?
.filter(col("name").is_not_null())?
.collect()
.await?;
let pretty_results = arrow::util::pretty::pretty_format_batches(&results)?.to_string();
println!("{}", pretty_results);
Ok(())
}
```
### Expected behavior
I was expecting the output to look like this:
```
+----+------+
| id | name |
+----+------+
| 2 | bob |
+----+------+
```
But the full dataset is returned instead:
```
+----+------+
| id | name |
+----+------+
| 1 | |
| 2 | bob |
+----+------+
```
### Additional context
Tested on v41.0.0
Replace `DataType::Dictionary(Box::new(DataType::UInt8), Box::new(DataType::Utf8))` with `DataType::Utf8` and it works.
Contributor guide
Assessment
This issue has not been assessed yet.