Better Grouping / aggregation pushdown
- Vorherrschende Sprache
- Rust
- Sterne
- 9.3k
- Forks
- 2.4k
- Ø Merge
- 3 T. 11 Std.
- Gemergte PRs (30 T.)
- 360
Beschreibung
### Is your feature request related to a problem or challenge?
@devinjdangelo asked on slack
https://the-asf.slack.com/archives/C04RJ0C85UZ/p1703891500973189?thread_ts=1703891392.037839&cid=C04RJ0C85UZ
> Related, is there any mechanism to push partial aggregations down to a table provider? E.g. “select count(*) from custom_provider” would push the aggregate expression down to the TableProvider rather than doing a scan.
Pushing grouping down in a plan is a classic analytic database optimization.
Among other things, it would allow doing fast data exploration like looking at counts / mins / maxes in different columns from only metadata.
### Describe the solution you'd like
I would like someone to propose
1. a change to `TableProvider` that would allow pushing aggregation down into the `TableProider`
2. A basic optimizer pass that pushes aggregation down when possible
We shouldn't innovate on API here, but should look at what other engines support in this area (e.g. TRINO / Spark / Postgres) and follow them unless there is a good reason to do something different
A good initial proof of concept would be to push `COUNT(*)`, `MIN(col)`, `MAX(col)` type queries down into parquet or other table providers that can provide such values from statistics or metadata with minimal work / data fetch
For example
```sql
datafusion-cli -c "select count(*), min(bigint_col), max(bigint_col) from './parquet-testing/data/alltypes_tiny_pages.parquet';"
DataFusion CLI v34.0.0
+----------+--------------------------------------------------------------------+--------------------------------------------------------------------+
| COUNT(*) | MIN(./parquet-testing/data/alltypes_tiny_pages.parquet.bigint_col) | MAX(./parquet-testing/data/alltypes_tiny_pages.parquet.bigint_col) |
+----------+--------------------------------------------------------------------+--------------------------------------------------------------------+
| 7300 | 0 | 90 |
+----------+--------------------------------------------------------------------+--------------------------------------------------------------------+
1 row in set. Query took 0.007 seconds.
```
The optimizer framework should eventually extend to pushing more sophisticated groupings (like pushing grouping down below joins) but many
### Describe alternatives you've considered
I believe [Eager Aggregation and Lazy Aggregation ](https://www.vldb.org/conf/1995/P345.PDF) is the classic paper on the optimizer portion
### Additional context
Beitragsleitfaden
Rechercherichtung
Beginnen Sie mit der Überprüfung des TableProvider-Einstiegspunkts und des Optimizer-Frameworks und vergleichen Sie anschließend wie angefordert die Unterstützung für das Pushdown von Aggregationen in Trino, Spark und Postgres. Ein erster Meilenstein würde die Änderung an TableProvider und den Optimizer-Pass definieren, mit einem Proof of Concept für COUNT(*), MIN(col) und MAX(col) unter Verwendung von Parquet-Statistiken oder Metadaten.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- rust
- Bereich
- data-engineering, databases, performance
- Issue-Typ
- Feature
- Schwierigkeit
- 5/5
- Geschätzter Aufwand
- Über eine Woche
- Aktivitätsstatus
- Veraltet
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 30/100