NVIDIA / NVIDIA/cudf

[BUG] [cudf_polars] Single cluster creates rapidsmpf objects for each pdsh query iteration

Open
#21,802 0 comments 0 reactions 1 assignee Claimed by @nirandaperera View on GitHub
bug cudf-polars rapids-mpf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**

Currently, cudf-polars `single` cluster with `rapidsmpf` execution, creates rapidsmpf objects for every query iteration, where are in other clusters (`distributed`, and `spmd`), they are created only once during the full query loop, and reused for every query (with a new `Context` for every query).

**Steps/Code to reproduce bug**

**Expected behavior**
`Single` should follow the same behavior as other clusters.
call `rapidsmpf.intergrations.single.get_worker_context, setup_worker` to access the global worker context and use it with a new Context for each query.

**Environment overview (please complete the following information)**
- Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)]
- Method of cuDF install: [conda, Docker, or from source]
- If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**
Please run and paste the output of the `cudf/print_env.sh` script here, to gather any other relevant environment details

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.