dask / dask/dask-expr

Drop is translated into a Projection which creates issues for inconsistent partitions

Open
#781 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
89
Forks
26
PR merge metrics
No merged PRs in 30d

Description

see ``test_drop_meta_mismatch`` in dask/dask

drop currently is translated into a Projection which is based on meta, so if some partition don't have some of those columns then you will run into KeyErrors, this can potentially be problematic.

A potential workaround is to create it's own Expression class and let it inherit from Projection so that it is taken into account in all optimisations, I have to think a little bit more about this if this is really a good idea

Contributor guide

Open the contributing guide

Research direction

Start with test_drop_meta_mismatch in dask/dask and trace how drop is translated into a Projection. Investigate how inconsistent partition columns lead to KeyErrors, then evaluate the issue's proposed Expression subclass approach while preserving optimization behavior; done means the test passes without errors for mismatched partition metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.