stats sampling doesn't work properly in case of partition pruning
Open
- Dominant language
- Python
- Stars
- 89
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
The files used for sampling and the filtered files might get out of sync resulting in weird errors. see https://github.com/dask/dask-expr/pull/1029/files#r1568572292 for a crude fix that we should make better.
Contributor guide
Research direction
The issue provides no repository file names; start by inspecting the files changed in PR #1029 and its referenced review comment. Reproduce stats sampling with partition pruning and trace where the sampling and filtered-file lists diverge; done means the mismatch no longer produces the reported errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100