[BUG] [cudf_polars] When rapidsmpf fails on a OOM, rest of the iterations also fail
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
When running rapidsmpf pdsh queries, if a query iteration fails with a OOM, rest of the iterations also fail because it kills the dask worker. Likely that the OOM exception is not properly handled.
**Steps/Code to reproduce bug**
**Expected behavior**
Only the current iteration should fail and record the exception. Rest of the iterations and queries should be able to continue without a problem (they may OOM in later iterations). The dask worker/ rapidsmpf context should not be affected.
Open questions:
- Since the rapidsmpf buffer resource is reused, would there be any dangling spilled data from the failed query? Do we need to do any special cleaning up?
Contributor guide
Assessment
This issue has not been assessed yet.