apache / apache/datafusion

TPC-DS Q75 regression in DF 55

Open
#25,157 1 comment 0 reactions 0 assignees View on GitHub
bug regression
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

We noticed a significant regression in our benchmarks, notably for TPC-DS Q75.

### To Reproduce

The essence of the regression can be captured by this minimal example
```sql
> SELECT VERSION();
+--------------------------------------------+
| version() |
+--------------------------------------------+
| Apache DataFusion 55.0.0, aarch64 on macos |
+--------------------------------------------+
1 row(s) fetched.
Elapsed 0.013 seconds.

> COPY (
SELECT (v % 5)::int AS d_year,
(v % 1000)::int AS brand,
(v % 100)::int AS class,
(v % 10)::int AS cat,
(v % 997)::int AS manu,
(v % 97)::bigint AS cnt,
arrow_cast(v::double / 100.0, 'Decimal128(31, 15)') AS amt
FROM generate_series(1, 10000000) AS s(v)
ORDER BY d_year
) TO '/tmp/sorted.parquet' STORED AS PARQUET;

> CREATE EXTERNAL TABLE src STORED AS PARQUET LOCATION '/tmp/sorted.parquet'
WITH ORDER (d_year ASC);
+----------+
| count |
+----------+
| 10000000 |
+----------+
1 row(s) fetched.
Elapsed 0.743 seconds.

0 row(s) fetched.
Elapsed 0.003 seconds.

> SELECT count(*) FROM (
SELECT DISTINCT d_year, brand, class, cat, manu, cnt, amt FROM src
);
+----------+
| count(*) |
+----------+
| 10000000 |
+----------+
1 row(s) fetched.
Elapsed 4.025 seconds.
```

Crucially the regression can be toggled away by disabling the `enable_migration_aggregate` config
```sql
> SET datafusion.execution.enable_migration_aggregate = false;
0 row(s) fetched.
Elapsed 0.001 seconds.

> SELECT count(*) FROM (
SELECT DISTINCT d_year, brand, class, cat, manu, cnt, amt FROM src
);
+----------+
| count(*) |
+----------+
| 10000000 |
+----------+
1 row(s) fetched.
Elapsed 0.470 seconds.
```

### Expected behavior

Compare to default behavior on DF54 (prior to `enable_migration_aggregate` config)

```sql
> SELECT VERSION();
+--------------------------------------------+
| version() |
+--------------------------------------------+
| Apache DataFusion 54.1.0, aarch64 on macos |
+--------------------------------------------+
1 row(s) fetched.
Elapsed 0.013 seconds.

> CREATE EXTERNAL TABLE src STORED AS PARQUET LOCATION '/tmp/sorted.parquet'
WITH ORDER (d_year ASC);
0 row(s) fetched.
Elapsed 0.003 seconds.

> SELECT count(*) FROM (
SELECT DISTINCT d_year, brand, class, cat, manu, cnt, amt FROM src
);
+----------+
| count(*) |
+----------+
| 10000000 |
+----------+
1 row(s) fetched.
Elapsed 0.513 seconds.
```

### Additional context

Potentially related to https://github.com/apache/datafusion/issues/24980

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the SQL example with TPC-DS Q75-style data, comparing the default setting with enable_migration_aggregate=false and DataFusion 54.1.0. Read the related issue #24980 and trace the aggregate migration path; done means the default DataFusion 55 behavior no longer shows the reported regression.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.