Drop is translated into a Projection which creates issues for inconsistent partitions
- Dominant language
- Python
- Stars
- 89
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
see ``test_drop_meta_mismatch`` in dask/dask
drop currently is translated into a Projection which is based on meta, so if some partition don't have some of those columns then you will run into KeyErrors, this can potentially be problematic.
A potential workaround is to create it's own Expression class and let it inherit from Projection so that it is taken into account in all optimisations, I have to think a little bit more about this if this is really a good idea
Contributor guide
Research direction
Start with test_drop_meta_mismatch in dask/dask and trace how drop is translated into a Projection. Investigate how inconsistent partition columns lead to KeyErrors, then evaluate the issue's proposed Expression subclass approach while preserving optimization behavior; done means the test passes without errors for mismatched partition metadata.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100