Optimizer `push_down_projection` failed due to different schemas
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
The query `SELECT * EXCEPT (field) FROM 'test.parquet' WHERE field = 'field';` panicked when being executed against an empty parquet file, below is the error message given by `datafusion-cli`:
> The parquet file is empty in data, not schema, it has a field `field`.
```
push_down_projection
caused by
Internal error: Optimizer rule 'push_down_projection' failed, due to generate a different schema,
original schema: DFSchema { fields: [], metadata: {} },
new schema: DFSchema { fields: [DFField { qualifier: Some(Bare { table: "test.parquet" }), field: Field { name: "field", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } }], metadata: {} }.
```
### To Reproduce
1. Use the following rust code to create an empty parquet file, data is empty but the fields in schema is NOT empty.
```
fn main() {
let file = OpenOptions::new()
.write(true)
.create(true)
.open("test.parquet")
.unwrap();
let writer = ArrowWriter::try_new(
file,
Arc::new(Schema::new(vec![Field::new("field", DataType::Utf8, true)])),
None,
)
.unwrap();
writer.close().unwrap();
}
```
2. Run the following query in `datafuion-cli`
> Or a Rust program using the `datafusion` library, you will get the same result
```shell
$ ls -l test.parquet
.rw-r--r--@ 263 steve 9 Aug 13:40 test.parquet
$ datafusion-cli
❯ SELECT * EXCEPT (field) FROM 'test.parquet' WHERE field = 'field';
push_down_projection
caused by
Internal error: Optimizer rule 'push_down_projection' failed, due to generate a different schema, original schema: DFSchema { fields: [], metadata: {} }, new schema: DFSchema { fields: [DFField { qualifier: Some(Bare { table: "test.parquet" }), field: Field { name: "field", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } }], metadata: {} }. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
```
### Expected behavior
The query can be successfully executed.
### Additional context
* parquet library version: 43
* datafusion-cli version: 28.0.0
Contributor guide
Research direction
Reproduce the failure in datafusion-cli using the empty test.parquet file created with ArrowWriter and the SELECT * EXCEPT query. Start by tracing the push_down_projection optimizer rule and how it handles the empty parquet schema. Done means the query executes successfully without the original and new schema mismatch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100