apache / apache/datafusion

Better Grouping / aggregation pushdown

Offen
#8,699 7 Kommentare 1 Reaktion 0 zugewiesene Personen Auf GitHub ansehen
enhancement
Vorherrschende Sprache
Rust
Sterne
9.3k
Forks
2.4k
Ø Merge
3 T. 11 Std.
Gemergte PRs (30 T.)
360

Beschreibung

### Is your feature request related to a problem or challenge?

@devinjdangelo asked on slack
https://the-asf.slack.com/archives/C04RJ0C85UZ/p1703891500973189?thread_ts=1703891392.037839&cid=C04RJ0C85UZ

> Related, is there any mechanism to push partial aggregations down to a table provider? E.g. “select count(*) from custom_provider” would push the aggregate expression down to the TableProvider rather than doing a scan.

Pushing grouping down in a plan is a classic analytic database optimization.

Among other things, it would allow doing fast data exploration like looking at counts / mins / maxes in different columns from only metadata.

### Describe the solution you'd like

I would like someone to propose
1. a change to `TableProvider` that would allow pushing aggregation down into the `TableProider`
2. A basic optimizer pass that pushes aggregation down when possible

We shouldn't innovate on API here, but should look at what other engines support in this area (e.g. TRINO / Spark / Postgres) and follow them unless there is a good reason to do something different

A good initial proof of concept would be to push `COUNT(*)`, `MIN(col)`, `MAX(col)` type queries down into parquet or other table providers that can provide such values from statistics or metadata with minimal work / data fetch

For example
```sql
datafusion-cli -c "select count(*), min(bigint_col), max(bigint_col) from './parquet-testing/data/alltypes_tiny_pages.parquet';"
DataFusion CLI v34.0.0
+----------+--------------------------------------------------------------------+--------------------------------------------------------------------+
| COUNT(*) | MIN(./parquet-testing/data/alltypes_tiny_pages.parquet.bigint_col) | MAX(./parquet-testing/data/alltypes_tiny_pages.parquet.bigint_col) |
+----------+--------------------------------------------------------------------+--------------------------------------------------------------------+
| 7300 | 0 | 90 |
+----------+--------------------------------------------------------------------+--------------------------------------------------------------------+
1 row in set. Query took 0.007 seconds.
```

The optimizer framework should eventually extend to pushing more sophisticated groupings (like pushing grouping down below joins) but many

### Describe alternatives you've considered

I believe [Eager Aggregation and Lazy Aggregation ](https://www.vldb.org/conf/1995/P345.PDF) is the classic paper on the optimizer portion

### Additional context

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginnen Sie mit der Überprüfung des TableProvider-Einstiegspunkts und des Optimizer-Frameworks und vergleichen Sie anschließend wie angefordert die Unterstützung für das Pushdown von Aggregationen in Trino, Spark und Postgres. Ein erster Meilenstein würde die Änderung an TableProvider und den Optimizer-Pass definieren, mit einem Proof of Concept für COUNT(*), MIN(col) und MAX(col) unter Verwendung von Parquet-Statistiken oder Metadaten.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
rust
Bereich
data-engineering, databases, performance
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
30/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.