dask / dask/dask-expr

Missing index columns after upgrade to dask-expr

Open
#1,144 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
89
Forks
26
PR merge metrics
No merged PRs in 30d

Description

Not sure if this qualifies as bug report or not. But we recently bumped dask (2023.12.0 -> 2024.9.1). We are now running into quite a bit of crashes involving missing index keys (usually preceded by read parquet + a bunch of groupby agg/apply).

Here is an example error msg:
```KeyError: "None of [Index(['advertisable_eid', 'user', 'subset_id'], dtype='object')] are in the [columns]"```

If I step through the code, I can see where its failing. But if I dump out the columns, the dataframe spits out the right columns. Debugging is complicated by the fact that the job only crashes on distributed workload (for now).

Curiously, adding a few strategic `df = df.optimize()` right after the `groupby` operation mysteriously resolves the issue. I only accidentally came across the solution and have no idea why it helps.

I am filing it here because the crash logs are filled with calls to this module (esp the fuse step). If there are any tips or tricks to get more debugging traces out of the system, kindly share 🙏🏽

Contributor guide

Open the contributing guide

Research direction

Start by reducing the distributed workload involving read parquet followed by groupby aggregation or apply, and capture the failure around the fuse step. Compare the missing-column failure with and without df.optimize() after groupby, and document a reproducible case plus the conditions that explain why optimization changes the result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.