NVIDIA / NVIDIA/cudf

[BUG] cudf-polars thread-offload uses hardware-determined number of threads

Open
#22,379 1 comment 0 reactions 1 assignee Claimed by @wence- View on GitHub
bug cudf-polars
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**

In the cudf-polars executor we use `asyncio.to_thread` to multi-thread the coroutines. This uses the "default" executor from the current event loop. But we have no control over how many threads that uses.

We might imagine setting the threadpoolexecutor the user must provide as the thread pool for asyncio but that doesn't work because asyncio _shuts it down_ when the `asyncio.run(...)` call ends. Argh!

So I think we should implement our own `to_thread`-like thing.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.