apache / apache/datafusion-ballista

Ballista 53.0.0 — Unsupported data type in sort merge join comparator: Dictionary(UInt16, Utf8)

Open
#1,837 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Environment:**
- Ballista: 53.0.0
- DataFusion: 53.1.0
- Dataset: TPCDS SF1 (Parquet on S3)
- Cluster: 1 coordinator (`c5n.2xlarge`) + 11 workers (`c5n.2xlarge`)
- Scheduling: PushStaged

**Reference:** DataFusion main passes all of the queries listed below.

---

**Affects:** TPCDS SF1 Q6, Q47, Q57, Q70

**Exact error messages:**

Q6 (stage 13):
```
Error: Query failed: 500 Arrow error: External error: Execution error: Job S7j0bce failed:
Job failed due to stage 13 failed: Task failed due to runtime execution error:
DataFusionError(NotImplemented("Unsupported data type in sort merge join comparator: Dictionary(UInt16, Utf8)"))
```

Q47 (stage 20):
```
Error: Query failed: 500 Arrow error: External error: Execution error: Job tH108Ae failed:
Job failed due to stage 20 failed: Task failed due to runtime execution error:
DataFusionError(NotImplemented("Unsupported data type in sort merge join comparator: Dictionary(UInt16, Utf8)"))
```

Q57 (stage 20):
```
Error: Query failed: 500 Arrow error: External error: Execution error: Job FwCaoXB failed:
Job failed due to stage 20 failed: Task failed due to runtime execution error:
DataFusionError(NotImplemented("Unsupported data type in sort merge join comparator: Dictionary(UInt16, Utf8)"))
```

Q70 (stage 11):
```
Error: Query failed: 500 Arrow error: External error: Execution error: Job hd52nrG failed:
Job failed due to stage 11 failed: Task failed due to runtime execution error:
DataFusionError(NotImplemented("Unsupported data type in sort merge join comparator: Dictionary(UInt16, Utf8)"))
```

**Description:**

These queries involve sort-merge joins or sorts on string columns encoded as `Dictionary(UInt16, Utf8)` in the TPCDS Parquet files.

DataFusion main passes all four queries at the same SF1 dataset.

Contributor guide

Open the contributing guide

Research direction

Reproduce TPCDS SF1 queries Q6, Q47, Q57, and Q70 with Ballista 53.0.0 and DataFusion 53.1.0, then trace the sort-merge join comparator handling for Dictionary(UInt16, Utf8) values. Compare behavior with DataFusion main; done means all four queries complete successfully without the unsupported-data-type error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.