apache / apache/datafusion

Support multiple order aware aggregate functions in a query

Open
#8,582 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Today DataFusion supports three aggregate functions that can be "order aware": `ARRAY_AGG`, `FIRST_VALUE` and `LAST_VALUE`. This means that you can supply a `ORDER BY` clause to their argument, for example `FIRST_VALUE(x ORDER BY time)`.

Today, there be only one single order specified across ALL order aware aggregate functions

For example
```SQL
❯ create table t(x int, y int) as values (1, 1), (1, 2), (1, 1), (1, 4), (2, 20), (2, 10);;
0 rows in set. Query took 0.003 seconds.

❯ select x, first_value(x ORDER BY y) from t GROUP BY x;
+---+------------------+
| x | FIRST_VALUE(t.x) |
+---+------------------+
| 2 | 2 |
| 1 | 1 |
+---+------------------+
2 rows in set. Query took 0.004 seconds.

❯ select x, first_value(x ORDER BY y), first_value(x ORDER BY y DESC) from t GROUP BY x;
+---+------------------+-----------------+
| x | FIRST_VALUE(t.x) | LAST_VALUE(t.x) |
+---+------------------+-----------------+
| 1 | 1 | 1 |
| 2 | 2 | 2 |
+---+------------------+-----------------+
2 rows in set. Query took 0.004 seconds.

❯ select x, first_value(x ORDER BY y), first_value(x ORDER BY y DESC NULLS LAST) from t GROUP BY x;
This feature is not implemented: Conflicting ordering requirements in aggregate functions is not supported
```

### Describe the solution you'd like

There are a few designs proposed here: https://github.com/apache/arrow-datafusion/pull/8558#issuecomment-1862649886

We are working on a more detailed proposal

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading the designs proposed in PR 8558's issue comment and the more detailed proposal mentioned in this issue. Use the SQL examples as the acceptance cases: queries with different ORDER BY clauses for multiple order-aware aggregate functions should work without the conflicting-ordering error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.