dask / dask/dask-expr

Docs on how to transition to using dask-expr -- given that it's now the default for dataframes in the new dask release

Open
#968 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
89
Forks
26
PR merge metrics
No merged PRs in 30d

Description

**Describe the issue**:
Creating dataframes without dask-expr works, but now with it being the default in the latest release it fails. I don't see migration/documentation on what the behavior changes are.

**Minimal Complete Verifiable Example**:
Run https://github.com/DAGWorks-Inc/hamilton/tree/main/plugin_tests/h_dask with the latest libraries (change conftest.py to use dask-expr query planning).

The code that is failing is this class https://github.com/DAGWorks-Inc/hamilton/blob/main/hamilton/plugins/h_dask.py#L175 -- which has some custom stuff and assumptions about how dask used to behave.

When I set:
```python
dask.config.set({‘dataframe.query-planning’: False})
```
Everything works -- so I'm convinced it's dask-expr that's the issue.

**Anything else we need to know?**:
I posted https://dask.discourse.group/t/what-changed-in-the-latest-release-with-the-default-to-use-dask-expr/2597 too.

**Environment**:
- Dask version: latest
- Python version: 3.9
- Operating System: Linux & mac.
- Install method (conda, pip, source): pip

Contributor guide

Open the contributing guide

Research direction

Run the MCV at plugin_tests/h_dask with the latest libraries and inspect conftest.py's dask-expr query-planning setting. Read hamilton/plugins/h_dask.py around line 175 and compare behavior with dataframe.query-planning disabled. Done means documenting the relevant behavior changes and migration steps for users moving to dask-expr as the default.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.