apache / apache/datafusion-ballista
Ballista 53.0.0 — Unsupported data type in sort merge join comparator: Dictionary(UInt16, Utf8)
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Environment:**
- Ballista: 53.0.0
- DataFusion: 53.1.0
- Dataset: TPCDS SF1 (Parquet on S3)
- Cluster: 1 coordinator (`c5n.2xlarge`) + 11 workers (`c5n.2xlarge`)
- Scheduling: PushStaged
**Reference:** DataFusion main passes all of the queries listed below.
---
**Affects:** TPCDS SF1 Q6, Q47, Q57, Q70
**Exact error messages:**
Q6 (stage 13):
```
Error: Query failed: 500 Arrow error: External error: Execution error: Job S7j0bce failed:
Job failed due to stage 13 failed: Task failed due to runtime execution error:
DataFusionError(NotImplemented("Unsupported data type in sort merge join comparator: Dictionary(UInt16, Utf8)"))
```
Q47 (stage 20):
```
Error: Query failed: 500 Arrow error: External error: Execution error: Job tH108Ae failed:
Job failed due to stage 20 failed: Task failed due to runtime execution error:
DataFusionError(NotImplemented("Unsupported data type in sort merge join comparator: Dictionary(UInt16, Utf8)"))
```
Q57 (stage 20):
```
Error: Query failed: 500 Arrow error: External error: Execution error: Job FwCaoXB failed:
Job failed due to stage 20 failed: Task failed due to runtime execution error:
DataFusionError(NotImplemented("Unsupported data type in sort merge join comparator: Dictionary(UInt16, Utf8)"))
```
Q70 (stage 11):
```
Error: Query failed: 500 Arrow error: External error: Execution error: Job hd52nrG failed:
Job failed due to stage 11 failed: Task failed due to runtime execution error:
DataFusionError(NotImplemented("Unsupported data type in sort merge join comparator: Dictionary(UInt16, Utf8)"))
```
**Description:**
These queries involve sort-merge joins or sorts on string columns encoded as `Dictionary(UInt16, Utf8)` in the TPCDS Parquet files.
DataFusion main passes all four queries at the same SF1 dataset.
Contributor guide
Research direction
Reproduce TPCDS SF1 queries Q6, Q47, Q57, and Q70 with Ballista 53.0.0 and DataFusion 53.1.0, then trace the sort-merge join comparator handling for Dictionary(UInt16, Utf8) values. Compare behavior with DataFusion main; done means all four queries complete successfully without the unsupported-data-type error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100