apache / apache/datafusion

Parquet schema file type coercion functionality is not applied on all fields in nested schemas, only on the top-level fields

Closed
#25,192 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

- ATM, the Parquet file schema type coercion applies view transforms for example only on top-level fields, but not on nested fields.
- For some Parquet schemas, where we request a view transform for a nested field, this translated to an unmodified schema passed to the reader. Then, when we try to apply predicates on the actual data, the actual predicate execution needs a cast, because the schemas are different.

The issue is visible in integration for example with delta-rs.
1. Delta-rs - when `schema_force_view_types` is true, applies this flag recursively on nested schemas.
2. The recursively-modified schema is passed through the layers to datafusion, until `apply_file_schema_type_coercions`
3. But, `apply_file_schema_type_coercions` does not take into account modifications on the nested fields, onlt on the top-level fields.
4. So, the Parquet is opened without having view types on nested fields
5. Then, when we try for example to apply predicates - which ARE using view types, because that is what delta-rs knows it has - datafusion needs to cast the data to have view types, resulting in longer predicate evaluation.

### To Reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at apply_file_schema_type_coercions and trace how the recursively modified schema from delta-rs reaches DataFusion. Reproduce the nested-field case with schema_force_view_types enabled, then verify nested view types are preserved through Parquet opening and predicate evaluation no longer requires an extra cast.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.