Cannot cast `Large` Types to normal arrow types
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
I am reading a parquet table, where one parquet file contains large arrow types, and one parquet file with normal arrow types. Datafusion doesn't like this and spits out an error that it cannot map the file schema to the table schema.
This even happens when I provide ParquetReadOptions with a schema that contains the normal arrow types. PyArrow for example has no issue reading these mixed large/normal types parquet table.
```rust
Plan("Cannot cast file schema field struct of type Struct([Field { name: \"x\", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"y\", data_type: LargeUtf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]) to table schema field of type Struct([Field { name: \"x\", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"y\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }])")
```
https://github.com/delta-io/delta-rs/actions/runs/7045987181/job/19176836034?pr=1926
### To Reproduce
```rust
stream = context
.ctx
.read_parquet(locations, ParquetReadOptions::default().schema(&file_schema))
.await?;
.execute_stream()
.await?
.map_err(|err| {
ParquetError::General(format!("Z-order failed while scanning data: {:?}", err))
})
.boxed();
```
### Expected behavior
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start at the read_parquet call using ParquetReadOptions::schema(...), then trace the schema-mapping error for nested LargeUtf8 and Utf8 fields. Reproduce with the mixed parquet files described in the issue and verify that the scan succeeds when the table schema uses normal Arrow types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100