apache / apache/datafusion

Dictionary-encoded group keys pay 27-37% aggregation tax vs Utf8View in TPC-H (Q2/Q19/Q20), DF 54 & 55 both

Open
#24,822 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

## Summary

Group-by aggregation over `Dictionary(Int32, Utf8View)` group keys is measurably slower than the identical query over plain `Utf8View` columns, on both DataFusion 54.0.1 and 55.0.0 (rev-checked). On TPC-H SF=10 in-memory:

- Q2 (p_type/p_size keys, 2 passes): dictionary **-27%** vs Utf8View
- Q19 (ps_shipmode key): dictionary **-37%**
- Q20 (s_name key): dictionary **-33%**

Reproduced with a self-contained bench: same RecordBatches, only the group-key column encoding differs (via Arrow cast before planning). No disk, no confounding.

## Repro sketch

```rust
// two sessions over the same batches; key column cast:
// a) Utf8View (baseline)
// b) Dictionary(Int32, Utf8View) (taxed)
let agg = ctx.sql("select l_returnflag, count(*) from t group by l_returnflag").await;
```

Timings (5-round median, single machine, Win11, DF 55.0.0):

| query | Utf8View | Dictionary | delta |
|---|---|---|---|
| Q2-like | 21.1ms | 28.9ms | **-27%** |
| Q19-like | 8.6ms | 13.6ms | **-37%** |
| Q20-like | 12.4ms | 18.5ms | **-33%** |

## Why this matters

Dictionary encoding is the natural choice for low-cardinality string columns
(exact-dictionary scans, smaller memory). But as soon as such a column is a
group key, users pay a ~1.3-1.6x aggregation penalty — the opposite of what
encoding choice should do to query time. In our columnar engine (custom
TableProvider over PAX storage) this forces a lose-lose choice between scan
speed and aggregation speed on every low-cardinality string column.

## Suspect area

Group-aggregation kernels appear to upcast/normalize dictionary keys per
batch rather than grouping on the codes directly (for single-batch
aggregations the dictionary *is* already a perfect group id in the common
no-duplicate case). Happy to share the standalone bench if useful.

DF 54.0.1 and 55.0.0 both affected; can test main on request.

## Environment

DataFusion 55.0.0 / 54.0.1, Arrow 57/56, Windows 11, rustc 1.93-nightly,
in-memory RecordBatches (8192-row batches), single-threaded execution to
isolate kernel cost.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the self-contained benchmark described in the issue, comparing Utf8View and Dictionary(Int32, Utf8View) group keys for the Q2-like, Q19-like, and Q20-like aggregations. Then inspect the group-aggregation kernels for the suspected per-batch dictionary normalization. Done means the dictionary-key aggregation no longer shows the reported 27–37% regression, with the benchmark measurements rerun on the affected setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.