[FEA] Support "dataframe.query-planning" config in ``dask.dataframe``
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
## PSA
To unblock CI failures related to the dask-expr migration, down-stream RAPIDS libraries can set the following environment variable in CI (before `dask.dataframe`/`dask_cudf` is ever imported):
```
export DASK_DATAFRAME__QUERY_PLANNING=False
```
**If you do this, please be sure to comment on the change, and link it to this meta issue.** (So I can make the necessary changes/fixes, and turn query-planning back on)
____
## Background
The `2024.2.0` release of Dask has deprecated the "legacy" `dask.dataframe` API. Given that dask-cudf (and much of RAPIDS) is tightly integrated with `dask.dataframe`, it is critical that `dask_cudf` be updated to use the new `dask_expr` backend smoothly.
Most of the heavy lifting is already being done in https://github.com/rapidsai/cudf/pull/14805. However, there will also be some follow-up work to expand coverage/examples/documentation/benchmarks. We will also need to update dask-cuda/explicit-comms.
## Action Items
**Basics** (to be covered by https://github.com/rapidsai/cudf/pull/14805):
- [x] Add dask-expr `DataFrameBackendEntrypoint` entrypoint for "cudf"
- [x] Align top-level `dask_cudf` imports with `dask.dataframe` for `"dataframe.query-planning"` support
**Expected Follow-up**:
- [x] Add `read_json` support (https://github.com/rapidsai/cudf/pull/15408)
- [x] Add `read_orc` support (https://github.com/rapidsai/cudf/pull/15439)
- [ ] `read_parquet` should always return DataFrame (not currently the case in dask-expr if `columns=`)
- [ ] Remove outdated `check_file_size` functionality from `dask_cudf.read_parquet`
- [x] Add s3 testing/support (https://github.com/rapidsai/cudf/pull/15408)
- [x] Add `read_text` support (https://github.com/rapidsai/cudf/pull/15439)
- [x] Fix unexplained test failures for categorical accessors (https://github.com/rapidsai/cudf/pull/15591)
- [x] Deprecate `to_dask_dataframe` API in favor of `to_backend` (https://github.com/rapidsai/cudf/pull/15592)
- [x] Deprecate `set_index(..., divisions="quantile")` (https://github.com/rapidsai/cudf/pull/15804)
- [x] Add `describe` support (seems to be working now? Just need to remove `xfail` markers)
- [x] Add `groupby` "collect" support (https://github.com/rapidsai/cudf/pull/15593)
- [ ] (Maybe?) add `as_index` support to `groupby`
- [x] Fix `get_dummy` support (https://github.com/dask/dask-expr/pull/1053)
- [ ] Fix sorting by categorical columns (https://github.com/rapidsai/cudf/pull/15701; **Related Issues**: https://github.com/rapidsai/cudf/issues/15641 & https://github.com/dask/dask/issues/11090 & https://github.com/rapidsai/cudf/pull/15801)
- [x] Fix sorting with nulls (https://github.com/rapidsai/cudf/pull/15639)
- [ ] `leftanti` merge support (Likely an error message in 24.06 and support in 24.08+)
- [x] `to_datetime` support (https://github.com/dask/dask-expr/pull/1035)
- [x] Add `melt` support (https://github.com/dask/dask-expr/pull/1049 & https://github.com/dask/dask/pull/11088)
**cuDF / Dask cuDF doc build**:
- [x] Revert https://github.com/rapidsai/cudf/pull/15343 (see: https://github.com/rapidsai/cudf/pull/15347)
**cuML support**:
- [x] [General debugging PR](https://github.com/rapidsai/cuml/pull/5835)
**cuxfilter support**:
- [x] (https://github.com/rapidsai/cuxfilter/pull/583, https://github.com/rapidsai/cuxfilter/pull/593)
**cugraph support**:
- [x] [General debugging PR](https://github.com/rapidsai/cugraph/pull/4325)
**Dask CUDA**:
- [ ] Explicit comms support (https://github.com/rapidsai/dask-cuda/issues/1311)
**Dask SQL**:
- [ ] Migrate predicate pushdown to dask-expr
**NeMo Curator**:
- [ ] Migrate custom-graph code and test against latest dask/cudf
**Merlin**:
- [ ] Port Merlin/NVTabular (Heavy lift - Aiming for 24.08)
Contributor guide
Assessment
This issue has not been assessed yet.