huggingface / huggingface/candle
Is there a roadmap or intention to support CUDA Graph?
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
vLLM v1 uses CUDA Graph to capture the execution workflow of the entire model, resulting in significant performance improvements compared to the previous version. I'm wondering if there are any plans to support CUDA Graph in Candle. Would it be possible to add `start_capture`, `end_capture`, and `replay` to the `Module` so that the captured graph can be replayed within the forward method? @LaurentMazare
Eric may also be interested in this @EricLBuehler
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing Candle's Module abstraction and the forward method, which are the entry points named in the issue. Determine whether CUDA Graph capture and replay fit those interfaces, then clarify the required API and scope; the issue does not name files or tests, and completion is not yet defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100