apache / apache/datafusion

Schema error after loading parquet stored with datafusion.execution.keep_partition_by_columns = TRUE

Open
#17,420 1 comment 0 reactions 1 assignee Claimed by @ShayanGho View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

When reading a parquet hive that was stored with `datafusion.execution.keep_partition_by_columns = TRUE`, the created table has two columns with the same name, raising a `Schema error: Schema contains duplicate qualified field name table.`

### To Reproduce

Create sample table with at least a `group` column to partition by.
Using datafusion-cli:
```SET datafusion.execution.keep_partition_by_columns = TRUE
COPY table TO 'test'
STORED AS PARQUET
PARTITION BY (group)
```
then
```
CREATE EXTERNAL TABLE test2
STORED AS PARQUET
PARTITIONED BY (group)
LOCATION 'test';

SELECT * FROM test2;
```

### Expected behavior

I guess in a session with `datafusion.execution.keep_partition_by_columns = TRUE` it makes sense to drop one of those two columns.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.