apache / apache/datafusion

In general the multi group by option seems to make certain scenarios worse.

Open
#17,850 1 comment 0 reactions 1 assignee Claimed by @nathanb9 View on GitHub
enhancement performance
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

In general the multi group by option seems to make certain scenarios worse.
I added a flag for easy control of whether it's used in this commit: https://github.com/ashdnazg/datafusion/commit/6a0b13bc2a5f9305b38f9bd0fa87cbf865e9c37b (see [enable-gby.patch](https://github.com/user-attachments/files/22625294/enable-gby.patch) for patch)
and then checked:
```
create or replace table foo as select (random() * 4)::integer as int_val, (random() * 4)::integer as int_val2, (random() * 4)::integer as int_val3 from generate_series(1, 1000000000);
set datafusion.execution.enable_multi_group_by to false;
select int_val, int_val2, int_val3 from foo GROUP BY int_val, int_val2, int_val3;
set datafusion.execution.enable_multi_group_by to true;
select int_val, int_val2, int_val3 from foo GROUP BY int_val, int_val2, int_val3;
```

I get `Elapsed 1.037 seconds.` with the flag set to false and `Elapsed 1.382 seconds.` with the flag set to true.

Perhaps the logic for when to use it should be more careful than just "whenever it's supported". But I suspect this PR is not the right spot for that discussion.

_Originally posted by @ashdnazg in https://github.com/apache/datafusion/issues/17726#issuecomment-3347258929_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.