dask / dask/dask

Gather-pq-parts takes a while

Open
#10,598 2 comments 0 reactions 0 assignees View on GitHub
needs attention needs triage
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

I'm reading a large parquet dataset with 128 MiB files and finding that `gather-pq-parts` takes a while when first creating the dataset. I'm wondering under what conditions this is triggered. I tried looking through the code, but it was fairly complex.

@rjzamora any help?

Contributor guide

Open the contributing guide

Research direction

Start by tracing where `gather-pq-parts` runs during initial creation of a dataset, using the reported 128 MiB parquet files as the reproduction case. Compare dataset creation with and without the conditions that trigger it; done means documenting those conditions and explaining the reported delay.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.