apache / apache/datafusion-python
Queries on tables via `register_dataset()` much slower than `register_parquet()`
- Vorherrschende Sprache
- Python
- Sterne
- 604
- Forks
- 174
- Ø Merge
- 1 T. 7 Std.
- Gemergte PRs (30 T.)
- 4
Beschreibung
**Describe the bug**
Queries against tables registered with `register_dataset()` perform around 80x slower than those registered with `register_parquet()`.
**To Reproduce**
```
import datafusion
import pyarrow.dataset as ds
from pathlib import Path
ctx = datafusion.SessionContext()
ctx.register_parquet("mytable", "*.parquet")
ctx.register_dataset("mytable2", ds.dataset(list(Path(".").glob("*.parquet"))))
```
Fast:
```
%time ctx.sql('select file_date, sum("Price" * "Volume") from mytable group by file_date order by file_date').to_arrow_table()
CPU times: user 2min 41s, sys: 3.35 s, total: 2min 45s
Wall time: 2.49 s
```
Slow:
```
%time ctx.sql('select file_date, sum("Price" * "Volume") from mytable2 group by file_date order by file_date').to_arrow_table()
CPU times: user 10min 51s, sys: 5min 40s, total: 16min 31s
Wall time: 3min 18s
```
**Expected behavior**
I'd expect these to be similar performance.
**Additional context**
The reason I'm using `ds.dataset` is because the actual files I'm interesting in accessing are not conveniently globbable (they're across multiple directories). So ideally I'd be able to provide a list of files to `ctx.register_parquet()` instead of a simple glob.
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Rechercherichtung
Start by running the reproduction comparing ctx.register_parquet() and ctx.register_dataset() with the grouped query, then trace the two registration entry points in the Python bindings. Done means the same parquet files have comparable query performance and register_parquet() can accept a list of files, as requested.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- python
- Bereich
- data-engineering, performance
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Veraltet
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 35/100