NVIDIA / NVIDIA/cudf

[BUG] [cudf_polars] When rapidsmpf fails on a OOM, rest of the iterations also fail

Open
#21,960 0 comments 0 reactions 0 assignees View on GitHub
bug cudf-polars dask no-oom rapids-mpf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
When running rapidsmpf pdsh queries, if a query iteration fails with a OOM, rest of the iterations also fail because it kills the dask worker. Likely that the OOM exception is not properly handled.

**Steps/Code to reproduce bug**

**Expected behavior**
Only the current iteration should fail and record the exception. Rest of the iterations and queries should be able to continue without a problem (they may OOM in later iterations). The dask worker/ rapidsmpf context should not be affected.
Open questions:
- Since the rapidsmpf buffer resource is reused, would there be any dangling spilled data from the failed query? Do we need to do any special cleaning up?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.