Schema error after loading parquet stored with datafusion.execution.keep_partition_by_columns = TRUE
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
When reading a parquet hive that was stored with `datafusion.execution.keep_partition_by_columns = TRUE`, the created table has two columns with the same name, raising a `Schema error: Schema contains duplicate qualified field name table.`
### To Reproduce
Create sample table with at least a `group` column to partition by.
Using datafusion-cli:
```SET datafusion.execution.keep_partition_by_columns = TRUE
COPY table TO 'test'
STORED AS PARQUET
PARTITION BY (group)
```
then
```
CREATE EXTERNAL TABLE test2
STORED AS PARQUET
PARTITIONED BY (group)
LOCATION 'test';
SELECT * FROM test2;
```
### Expected behavior
I guess in a session with `datafusion.execution.keep_partition_by_columns = TRUE` it makes sense to drop one of those two columns.
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.