apache / apache/datafusion

Some aggregates silently ignore `IGNORE NULLS` and `ORDER BY` on arguments

Open
#9,924 8 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

DataFusion now supports providing ordering and nulls information to aggregates, but some aggregates ignore the flags silently

For example
```sql
❯ select first_value(column1 ORDER BY column2) FROM (values (1,2), (3,4), (-1,0)) ;
+----------------------+
| FIRST_VALUE(column1) |
+----------------------+
| -1 |
+----------------------+

❯ select first_value(column1 ORDER BY column2) IGNORE NULLS FROM (values (1,2), (3,4), (null,0)) ;
+----------------------+
| FIRST_VALUE(column1) |
+----------------------+
| 1 |
+----------------------+
1 row in set. Query took 0.002 seconds.
```

### To Reproduce

```sql
❯ select count(*) from (values (1), (null), (2));
+----------+
| COUNT(*) |
+----------+
| 3 |
+----------+
1 row in set. Query took 0.039 seconds.

❯ select count(*) IGNORE NULLS from (values (1), (null), (2));
+----------+
| COUNT(*) |
+----------+
| 3 |
+----------+
1 row in set. Query took 0.001 seconds.
```

Also, for ordering

```sql
❯ select avg(column1 ORDER BY column2) FROM (values (1,2), (3,4), (null,0)) ;
+--------------+
| AVG(column1) |
+--------------+
| 2.0 |
+--------------+
1 row in set. Query took 0.008 seconds.
```

### Expected behavior

I expect

```sql
❯ select count(*) IGNORE NULLS from (values (1), (null), (2));
```

To error with "IGNORE NULLS is not supported

I also expect

```sql
❯ select avg(column1 ORDER BY column2) FROM (values (1,2), (3,4), (null,0)) ;
```

to error with "ORDER BY" not supported for avg

### Additional context

@jayzhan211 has some good ideas at https://github.com/apache/arrow-datafusion/pull/9920#discussion_r1549603208 about how to make checking this easier / harder to miss

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the COUNT and AVG SQL examples in DataFusion, then inspect the aggregate handling that processes IGNORE NULLS and argument ORDER BY. Compare the behavior with the discussion linked from pull request 9920. Done means unsupported combinations produce explicit errors instead of being silently ignored.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.