NVIDIA / NVIDIA/cudf

`num_py_executors` thread pool visibility

Open
#23,814 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

`ir_context.to_thread(...)` (`dsl/ir.py`) dispatches blocking/GPU-launching work onto a `ThreadPoolExecutor` sized by `num_py_executors` (default 8, constructed the same way in `engine/spmd.py`, `engine/dask.py`, and `engine/ray.py`). It's not I/O-specific: `read_chunk` uses it, but so does join-partition compute (`actor_graph/join.py`), window/`Over` evaluation (`actor_graph/over.py`), generic per-chunk `do_evaluate` for other IR node types (`actor_graph/nodes.py`), shared per-chunk evaluation/concat helpers used by groupby and sort (`actor_graph/utils.py`), and AllGather result concatenation (`actor_graph/collectives/allgather.py`). It's one process-wide pool shared by every actor type in the query, not just scan nodes. There's no visibility into whether it's saturated or idle today.

This is a distinct layer from both rapidsai/kvikio#1044, which executes byte-range fetches inside a `read_chunk` call specifically, and rapidsai/rapidsmpf#1169, which resumes the coroutine that *calls* `to_thread` in the first place. An I/O task can be slow to start for any of three separate reasons stacked on top of each other:

- its coroutine isn't being resumed
- its coroutine is resumed but has to wait for a free `num_py_executors` thread (now contended by every other actor type too, not just other scans)
- it gets a thread immediately but the read itself is slow

Right now none of the three are distinguishable. This sub-issue closes the second one. The first is closed by rapidsai/rapidsmpf#1169, and the third by #23811 and #23812, which already split `read_chunk`'s own timing into admission-wait vs. execution time once a thread is actually running it.

Add the same kind of queue/running count Python's `ThreadPoolExecutor` doesn't expose natively (it only exposes `_work_queue.qsize()` as a private implementation detail, worth wrapping rather than relying on that directly).

Contributor guide

Open the contributing guide

Research direction

Start in dsl/ir.py at ir_context.to_thread, then compare ThreadPoolExecutor construction in engine/spmd.py, engine/dask.py, and engine/ray.py. Review the named actor_graph call sites to understand the shared pool's usage. Done means exposing queue and running counts so thread-wait contention can be distinguished from coroutine scheduling and read execution time.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.