apache / apache/datafusion

Cannot cast `Large` Types to normal arrow types

Open
#8,372 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

I am reading a parquet table, where one parquet file contains large arrow types, and one parquet file with normal arrow types. Datafusion doesn't like this and spits out an error that it cannot map the file schema to the table schema.

This even happens when I provide ParquetReadOptions with a schema that contains the normal arrow types. PyArrow for example has no issue reading these mixed large/normal types parquet table.

```rust
Plan("Cannot cast file schema field struct of type Struct([Field { name: \"x\", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"y\", data_type: LargeUtf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]) to table schema field of type Struct([Field { name: \"x\", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"y\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }])")
```

https://github.com/delta-io/delta-rs/actions/runs/7045987181/job/19176836034?pr=1926

### To Reproduce

```rust
stream = context
.ctx
.read_parquet(locations, ParquetReadOptions::default().schema(&file_schema))
.await?;
.execute_stream()
.await?
.map_err(|err| {
ParquetError::General(format!("Z-order failed while scanning data: {:?}", err))
})
.boxed();
```

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at the read_parquet call using ParquetReadOptions::schema(...), then trace the schema-mapping error for nested LargeUtf8 and Utf8 fields. Reproduce with the mixed parquet files described in the issue and verify that the scan succeeds when the table schema uses normal Arrow types.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.