cupy / cupy/cupy

Improve stream capture support

Open
#6,290 2 comments 2 reactions 0 assignees View on GitHub
cat:enhancement prio:medium
Dominant language
Python
Stars
12.3k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
45

Description

Context: #4567 added basic stream capture support but disabled calls to almost all CUDA libraries during capture (except for cuFFT, cuTENSOR, cuSPARSELt, NCCL). This meta-issue is created to track **CuPy's support** for each CUDA library. (The support status of each library below is recorded as of CUDA 11.5.1.)

- [ ] cuBLAS: Most of the routines support stream capture, but some checks (like pointer mode) are required, see https://docs.nvidia.com/cuda/cublas/index.html#CUDA-graphs.
- [ ] cuSOLVER: Does not support stream capture
- [ ] cuRAND: Does not support stream capture
- [ ] cuFFT: ~~[Fully supported](https://docs.nvidia.com/cuda/cufft/index.html#cuda-graphs-support)~~ **Update**: see https://github.com/cupy/cupy/issues/6290#issuecomment-1114357508
- [ ] cuSPARSE: Most of the routines support stream capture but some require the usage of cudaMallocAsync, see https://docs.nvidia.com/cuda/cusparse/index.html#optimization-notes
- [ ] cuSPARSELt: Some routines support capture, see the API reference at https://docs.nvidia.com/cuda/cusparselt/functions.html?highlight=capture
- [x] cuTENSOR: All routines support capture except for autotuning that CuPy does not expose, see https://docs.nvidia.com/cuda/cutensor/user_guide.html#cuda-graph-support
- [ ] NCCL: Supported since NCCL 2.9, but with some caveats (ex: the capture mode), see https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/cudagraph.html
- [ ] cuDNN: The documentation mentions no stream capture, so assumed no support

In terms of CuPy's handling of stream capture, some room for potential improvements:
- [ ] Reduce the (tiny) overhead of checking the capture status (see https://github.com/cupy/cupy/pull/4567#issuecomment-1004555831)
- [ ] Change `is_capturing` from a method to a property? (See https://github.com/cupy/cupy/pull/4567#discussion_r778523958)

Contributor guide

Open the contributing guide

Research direction

Start by reading the stream-capture work in #4567 and the linked CUDA documentation for each library in the checklist. Determine which library or capture-handling improvement is still actionable, then define completion around implementing and testing that specific support; this meta-issue does not identify files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.