NVIDIA / NVIDIA/cudf

[BUG][cudf_polars] distributed cluster creates a new `Context` for every query run

Open
#21,989 7 comments 0 reactions 1 assignee Claimed by @nirandaperera View on GitHub
bug cudf-polars
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**

https://github.com/rapidsai/cudf/blob/main/python/cudf_polars/cudf_polars/experimental/rapidsmpf/dask.py#L173-L183

Dask cluster mode creates a new context for every query run. This also creates a new Options object which is different to the Options object used to create BufferResource.

**Steps/Code to reproduce bug**

**Expected behavior**
Since we dont use tasks anymore, we should create a rapidsmpf Context and stash it in the Dask worker context similar to BufferResource and reuse it for every query

**Environment overview (please complete the following information)**
- Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)]
- Method of cuDF install: [conda, Docker, or from source]
- If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**
Please run and paste the output of the `cudf/print_env.sh` script here, to gather any other relevant environment details

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.