NVIDIA / NVIDIA/cudf

[FEA] Support "dataframe.query-planning" config in ``dask.dataframe``

Open
#15,027 0 comments 0 reactions 1 assignee Claimed by @rjzamora View on GitHub
dask feature request
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

## PSA

To unblock CI failures related to the dask-expr migration, down-stream RAPIDS libraries can set the following environment variable in CI (before `dask.dataframe`/`dask_cudf` is ever imported):

```
export DASK_DATAFRAME__QUERY_PLANNING=False
```

**If you do this, please be sure to comment on the change, and link it to this meta issue.** (So I can make the necessary changes/fixes, and turn query-planning back on)

____

## Background

The `2024.2.0` release of Dask has deprecated the "legacy" `dask.dataframe` API. Given that dask-cudf (and much of RAPIDS) is tightly integrated with `dask.dataframe`, it is critical that `dask_cudf` be updated to use the new `dask_expr` backend smoothly.

Most of the heavy lifting is already being done in https://github.com/rapidsai/cudf/pull/14805. However, there will also be some follow-up work to expand coverage/examples/documentation/benchmarks. We will also need to update dask-cuda/explicit-comms.

## Action Items

**Basics** (to be covered by https://github.com/rapidsai/cudf/pull/14805):
- [x] Add dask-expr `DataFrameBackendEntrypoint` entrypoint for "cudf"
- [x] Align top-level `dask_cudf` imports with `dask.dataframe` for `"dataframe.query-planning"` support

**Expected Follow-up**:
- [x] Add `read_json` support (https://github.com/rapidsai/cudf/pull/15408)
- [x] Add `read_orc` support (https://github.com/rapidsai/cudf/pull/15439)
- [ ] `read_parquet` should always return DataFrame (not currently the case in dask-expr if `columns=`)
- [ ] Remove outdated `check_file_size` functionality from `dask_cudf.read_parquet`
- [x] Add s3 testing/support (https://github.com/rapidsai/cudf/pull/15408)
- [x] Add `read_text` support (https://github.com/rapidsai/cudf/pull/15439)
- [x] Fix unexplained test failures for categorical accessors (https://github.com/rapidsai/cudf/pull/15591)
- [x] Deprecate `to_dask_dataframe` API in favor of `to_backend` (https://github.com/rapidsai/cudf/pull/15592)
- [x] Deprecate `set_index(..., divisions="quantile")` (https://github.com/rapidsai/cudf/pull/15804)
- [x] Add `describe` support (seems to be working now? Just need to remove `xfail` markers)
- [x] Add `groupby` "collect" support (https://github.com/rapidsai/cudf/pull/15593)
- [ ] (Maybe?) add `as_index` support to `groupby`
- [x] Fix `get_dummy` support (https://github.com/dask/dask-expr/pull/1053)
- [ ] Fix sorting by categorical columns (https://github.com/rapidsai/cudf/pull/15701; **Related Issues**: https://github.com/rapidsai/cudf/issues/15641 & https://github.com/dask/dask/issues/11090 & https://github.com/rapidsai/cudf/pull/15801)
- [x] Fix sorting with nulls (https://github.com/rapidsai/cudf/pull/15639)
- [ ] `leftanti` merge support (Likely an error message in 24.06 and support in 24.08+)
- [x] `to_datetime` support (https://github.com/dask/dask-expr/pull/1035)
- [x] Add `melt` support (https://github.com/dask/dask-expr/pull/1049 & https://github.com/dask/dask/pull/11088)

**cuDF / Dask cuDF doc build**:
- [x] Revert https://github.com/rapidsai/cudf/pull/15343 (see: https://github.com/rapidsai/cudf/pull/15347)

**cuML support**:
- [x] [General debugging PR](https://github.com/rapidsai/cuml/pull/5835)

**cuxfilter support**:
- [x] (https://github.com/rapidsai/cuxfilter/pull/583, https://github.com/rapidsai/cuxfilter/pull/593)

**cugraph support**:
- [x] [General debugging PR](https://github.com/rapidsai/cugraph/pull/4325)

**Dask CUDA**:
- [ ] Explicit comms support (https://github.com/rapidsai/dask-cuda/issues/1311)

**Dask SQL**:
- [ ] Migrate predicate pushdown to dask-expr

**NeMo Curator**:
- [ ] Migrate custom-graph code and test against latest dask/cudf

**Merlin**:
- [ ] Port Merlin/NVTabular (Heavy lift - Aiming for 24.08)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.