Docs on how to transition to using dask-expr -- given that it's now the default for dataframes in the new dask release
- Dominant language
- Python
- Stars
- 89
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the issue**:
Creating dataframes without dask-expr works, but now with it being the default in the latest release it fails. I don't see migration/documentation on what the behavior changes are.
**Minimal Complete Verifiable Example**:
Run https://github.com/DAGWorks-Inc/hamilton/tree/main/plugin_tests/h_dask with the latest libraries (change conftest.py to use dask-expr query planning).
The code that is failing is this class https://github.com/DAGWorks-Inc/hamilton/blob/main/hamilton/plugins/h_dask.py#L175 -- which has some custom stuff and assumptions about how dask used to behave.
When I set:
```python
dask.config.set({‘dataframe.query-planning’: False})
```
Everything works -- so I'm convinced it's dask-expr that's the issue.
**Anything else we need to know?**:
I posted https://dask.discourse.group/t/what-changed-in-the-latest-release-with-the-default-to-use-dask-expr/2597 too.
**Environment**:
- Dask version: latest
- Python version: 3.9
- Operating System: Linux & mac.
- Install method (conda, pip, source): pip
Contributor guide
Research direction
Run the MCV at plugin_tests/h_dask with the latest libraries and inspect conftest.py's dask-expr query-planning setting. Read hamilton/plugins/h_dask.py around line 175 and compare behavior with dataframe.query-planning disabled. Done means documenting the relevant behavior changes and migration steps for users moving to dask-expr as the default.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100