dask / dask/dask-expr

Reading a list of S3 parquet files with query planning enabled is ~25x slower

Open
#1,061 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
89
Forks
26
PR merge metrics
No merged PRs in 30d

Description

Was struggling to understand why creating a dask dataframe from a large list of parquet files was taking ages. Eventually tried disabling query planning and saw normal timing again. These are all relatively small S3 files ~1MB. There is no metadata file or similar.

![Screenshot 2024-05-10 at 3 21 23 PM](https://github.com/dask/dask-expr/assets/149107659/b32f7ca5-7653-4e66-9839-480f0f2cd286)

**Environment**:
- dask==2024.5.0
- dask-expr==1.1.0
- python==3.10

Contributor guide

Open the contributing guide

Research direction

No source files, tests, or entry points are named. Start by reproducing dataframe creation with query planning enabled and disabled using the reported small S3 parquet files, then trace where the planning time is spent. Done means the large timing difference is resolved or clearly characterized with a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.