apache / apache/datafusion

Better Grouping / aggregation pushdown

Aperta
#8,699 7 commenti 1 reazione 0 assegnatari Vedi su GitHub
enhancement
Lingua principale
Rust
Stelle
9.3k
Fork
2.4k
Merge medio
3g 11h
PR unite (30g)
360

Descrizione

### Is your feature request related to a problem or challenge?

@devinjdangelo asked on slack
https://the-asf.slack.com/archives/C04RJ0C85UZ/p1703891500973189?thread_ts=1703891392.037839&cid=C04RJ0C85UZ

> Related, is there any mechanism to push partial aggregations down to a table provider? E.g. “select count(*) from custom_provider” would push the aggregate expression down to the TableProvider rather than doing a scan.

Pushing grouping down in a plan is a classic analytic database optimization.

Among other things, it would allow doing fast data exploration like looking at counts / mins / maxes in different columns from only metadata.

### Describe the solution you'd like

I would like someone to propose
1. a change to `TableProvider` that would allow pushing aggregation down into the `TableProider`
2. A basic optimizer pass that pushes aggregation down when possible

We shouldn't innovate on API here, but should look at what other engines support in this area (e.g. TRINO / Spark / Postgres) and follow them unless there is a good reason to do something different

A good initial proof of concept would be to push `COUNT(*)`, `MIN(col)`, `MAX(col)` type queries down into parquet or other table providers that can provide such values from statistics or metadata with minimal work / data fetch

For example
```sql
datafusion-cli -c "select count(*), min(bigint_col), max(bigint_col) from './parquet-testing/data/alltypes_tiny_pages.parquet';"
DataFusion CLI v34.0.0
+----------+--------------------------------------------------------------------+--------------------------------------------------------------------+
| COUNT(*) | MIN(./parquet-testing/data/alltypes_tiny_pages.parquet.bigint_col) | MAX(./parquet-testing/data/alltypes_tiny_pages.parquet.bigint_col) |
+----------+--------------------------------------------------------------------+--------------------------------------------------------------------+
| 7300 | 0 | 90 |
+----------+--------------------------------------------------------------------+--------------------------------------------------------------------+
1 row in set. Query took 0.007 seconds.
```

The optimizer framework should eventually extend to pushing more sophisticated groupings (like pushing grouping down below joins) but many

### Describe alternatives you've considered

I believe [Eager Aggregation and Lazy Aggregation ](https://www.vldb.org/conf/1995/P345.PDF) is the classic paper on the optimizer portion

### Additional context

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia esaminando l’entry point di TableProvider e il framework dell’optimizer, quindi confronta, come richiesto, il supporto all’aggregation pushdown in Trino, Spark e Postgres. Un primo traguardo definirebbe la modifica a TableProvider e il pass dell’optimizer, con una proof of concept per COUNT(*), MIN(col) e MAX(col) usando statistiche o metadati Parquet.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
rust
Ambito
data-engineering, databases, performance
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
30/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.