Reading a list of S3 parquet files with query planning enabled is ~25x slower
- Dominant language
- Python
- Stars
- 89
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
Was struggling to understand why creating a dask dataframe from a large list of parquet files was taking ages. Eventually tried disabling query planning and saw normal timing again. These are all relatively small S3 files ~1MB. There is no metadata file or similar.

**Environment**:
- dask==2024.5.0
- dask-expr==1.1.0
- python==3.10
Contributor guide
Research direction
No source files, tests, or entry points are named. Start by reproducing dataframe creation with query planning enabled and disabled using the reported small S3 parquet files, then trace where the planning time is spent. Done means the large timing difference is resolved or clearly characterized with a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100