apache / apache/datafusion

Fail to merge schema field for partitioned table with dict

Open
#17,421 2 comments 0 reactions 1 assignee Claimed by @VaibhaveS View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

I am trying out datafusion for some refactoring. I am testing with the following setup:
```
+-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-------------+
| column_name | data_type | is_nullable |
+-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-------------+
| name | Utf8View | NO |
| group | Dictionary(UInt16, Utf8) | NO |
| market | Struct([Field { name: "market", data_type: Dictionary(UInt16, Utf8), nullable: true, dict_id: 1, dict_is_ordered: false, metadata: {} }, ...])| YES |
|... | ... | ... |
+-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-------------+
```

I am trying to use `group` as a partition key and storing the table as a parquet hive.
When reading the hive either with datafusion-cli or using Rust with
```
ctx.register_parquet(
"test",
src,
ParquetReadOptions::default()
.file_sort_order(vec![vec![
SortExpr::new(col("name"), true, false),
SortExpr::new(col("group"), true, false),
]])
.table_partition_cols(vec![
(
"group".to_string(),
DataType::Dictionary(Box::new(DataType::UInt16), Box::new(DataType::Utf8)),
),
]),
)
.await
```
I am getting the following error:
`Arrow error: Schema error: Fail to merge schema field 'market' because from dict_id = 1 does not match 0`.

I assume this is caused because the order of dict creation is different. Do note that this does not occur with `datafusion.execution.keep_partition_by_columns = True`. But then you run into https://github.com/apache/datafusion/issues/17420

### To Reproduce

Create table with schema from above.
```
COPY test TO 'test_out' STORED AS PARQUET PARTITIONED BY group;

CREATE EXTERNAL TABLE test2
STORED AS PARQUET
PARTITIONED BY (group)
LOCATION 'test_out';
```

### Expected behavior

The hive/partitioned table should be loaded.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.