apache / apache/datafusion

Optimizer `push_down_projection` failed due to different schemas

Open
#7,241 10 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

The query `SELECT * EXCEPT (field) FROM 'test.parquet' WHERE field = 'field';` panicked when being executed against an empty parquet file, below is the error message given by `datafusion-cli`:

> The parquet file is empty in data, not schema, it has a field `field`.

```
push_down_projection
caused by
Internal error: Optimizer rule 'push_down_projection' failed, due to generate a different schema,

original schema: DFSchema { fields: [], metadata: {} },

new schema: DFSchema { fields: [DFField { qualifier: Some(Bare { table: "test.parquet" }), field: Field { name: "field", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } }], metadata: {} }.
```

### To Reproduce

1. Use the following rust code to create an empty parquet file, data is empty but the fields in schema is NOT empty.

```
fn main() {
let file = OpenOptions::new()
.write(true)
.create(true)
.open("test.parquet")
.unwrap();
let writer = ArrowWriter::try_new(
file,
Arc::new(Schema::new(vec![Field::new("field", DataType::Utf8, true)])),
None,
)
.unwrap();

writer.close().unwrap();
}
```

2. Run the following query in `datafuion-cli`

> Or a Rust program using the `datafusion` library, you will get the same result

```shell
$ ls -l test.parquet
.rw-r--r--@ 263 steve 9 Aug 13:40 test.parquet

$ datafusion-cli
❯ SELECT * EXCEPT (field) FROM 'test.parquet' WHERE field = 'field';
push_down_projection
caused by
Internal error: Optimizer rule 'push_down_projection' failed, due to generate a different schema, original schema: DFSchema { fields: [], metadata: {} }, new schema: DFSchema { fields: [DFField { qualifier: Some(Bare { table: "test.parquet" }), field: Field { name: "field", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } }], metadata: {} }. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
```

### Expected behavior

The query can be successfully executed.

### Additional context

* parquet library version: 43
* datafusion-cli version: 28.0.0

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure in datafusion-cli using the empty test.parquet file created with ArrowWriter and the SELECT * EXCEPT query. Start by tracing the push_down_projection optimizer rule and how it handles the empty parquet schema. Done means the query executes successfully without the original and new schema mismatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.