Improve stream capture support
- Dominant language
- Python
- Stars
- 12.3k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 45
Description
Context: #4567 added basic stream capture support but disabled calls to almost all CUDA libraries during capture (except for cuFFT, cuTENSOR, cuSPARSELt, NCCL). This meta-issue is created to track **CuPy's support** for each CUDA library. (The support status of each library below is recorded as of CUDA 11.5.1.)
- [ ] cuBLAS: Most of the routines support stream capture, but some checks (like pointer mode) are required, see https://docs.nvidia.com/cuda/cublas/index.html#CUDA-graphs.
- [ ] cuSOLVER: Does not support stream capture
- [ ] cuRAND: Does not support stream capture
- [ ] cuFFT: ~~[Fully supported](https://docs.nvidia.com/cuda/cufft/index.html#cuda-graphs-support)~~ **Update**: see https://github.com/cupy/cupy/issues/6290#issuecomment-1114357508
- [ ] cuSPARSE: Most of the routines support stream capture but some require the usage of cudaMallocAsync, see https://docs.nvidia.com/cuda/cusparse/index.html#optimization-notes
- [ ] cuSPARSELt: Some routines support capture, see the API reference at https://docs.nvidia.com/cuda/cusparselt/functions.html?highlight=capture
- [x] cuTENSOR: All routines support capture except for autotuning that CuPy does not expose, see https://docs.nvidia.com/cuda/cutensor/user_guide.html#cuda-graph-support
- [ ] NCCL: Supported since NCCL 2.9, but with some caveats (ex: the capture mode), see https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/cudagraph.html
- [ ] cuDNN: The documentation mentions no stream capture, so assumed no support
In terms of CuPy's handling of stream capture, some room for potential improvements:
- [ ] Reduce the (tiny) overhead of checking the capture status (see https://github.com/cupy/cupy/pull/4567#issuecomment-1004555831)
- [ ] Change `is_capturing` from a method to a property? (See https://github.com/cupy/cupy/pull/4567#discussion_r778523958)
Contributor guide
Research direction
Start by reading the stream-capture work in #4567 and the linked CUDA documentation for each library in the checklist. Determine which library or capture-handling improvement is still actionable, then define completion around implementing and testing that specific support; this meta-issue does not identify files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100