Fail to merge schema field for partitioned table with dict
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
I am trying out datafusion for some refactoring. I am testing with the following setup:
```
+-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-------------+
| column_name | data_type | is_nullable |
+-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-------------+
| name | Utf8View | NO |
| group | Dictionary(UInt16, Utf8) | NO |
| market | Struct([Field { name: "market", data_type: Dictionary(UInt16, Utf8), nullable: true, dict_id: 1, dict_is_ordered: false, metadata: {} }, ...])| YES |
|... | ... | ... |
+-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-------------+
```
I am trying to use `group` as a partition key and storing the table as a parquet hive.
When reading the hive either with datafusion-cli or using Rust with
```
ctx.register_parquet(
"test",
src,
ParquetReadOptions::default()
.file_sort_order(vec![vec![
SortExpr::new(col("name"), true, false),
SortExpr::new(col("group"), true, false),
]])
.table_partition_cols(vec![
(
"group".to_string(),
DataType::Dictionary(Box::new(DataType::UInt16), Box::new(DataType::Utf8)),
),
]),
)
.await
```
I am getting the following error:
`Arrow error: Schema error: Fail to merge schema field 'market' because from dict_id = 1 does not match 0`.
I assume this is caused because the order of dict creation is different. Do note that this does not occur with `datafusion.execution.keep_partition_by_columns = True`. But then you run into https://github.com/apache/datafusion/issues/17420
### To Reproduce
Create table with schema from above.
```
COPY test TO 'test_out' STORED AS PARQUET PARTITIONED BY group;
CREATE EXTERNAL TABLE test2
STORED AS PARQUET
PARTITIONED BY (group)
LOCATION 'test_out';
```
### Expected behavior
The hive/partitioned table should be loaded.
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.