Gather-pq-parts takes a while
Open
needs attention
needs triage
- Dominant language
- Python
- Stars
- 13.9k
- Forks
- 2k
- PR merge metrics
- No merged PRs in 30d
Description
I'm reading a large parquet dataset with 128 MiB files and finding that `gather-pq-parts` takes a while when first creating the dataset. I'm wondering under what conditions this is triggered. I tried looking through the code, but it was fairly complex.
@rjzamora any help?
Contributor guide
Research direction
Start by tracing where `gather-pq-parts` runs during initial creation of a dataset, using the reported 128 MiB parquet files as the reproduction case. Compare dataset creation with and without the conditions that trigger it; done means documenting those conditions and explaining the reported delay.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100