Missing index columns after upgrade to dask-expr
- Dominant language
- Python
- Stars
- 89
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
Not sure if this qualifies as bug report or not. But we recently bumped dask (2023.12.0 -> 2024.9.1). We are now running into quite a bit of crashes involving missing index keys (usually preceded by read parquet + a bunch of groupby agg/apply).
Here is an example error msg:
```KeyError: "None of [Index(['advertisable_eid', 'user', 'subset_id'], dtype='object')] are in the [columns]"```
If I step through the code, I can see where its failing. But if I dump out the columns, the dataframe spits out the right columns. Debugging is complicated by the fact that the job only crashes on distributed workload (for now).
Curiously, adding a few strategic `df = df.optimize()` right after the `groupby` operation mysteriously resolves the issue. I only accidentally came across the solution and have no idea why it helps.
I am filing it here because the crash logs are filled with calls to this module (esp the fuse step). If there are any tips or tricks to get more debugging traces out of the system, kindly share 🙏🏽
Contributor guide
Research direction
Start by reducing the distributed workload involving read parquet followed by groupby aggregation or apply, and capture the failure around the fuse step. Compare the missing-column failure with and without df.optimize() after groupby, and document a reproducible case plus the conditions that explain why optimization changes the result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100