huggingface / huggingface/candle

Is there a roadmap or intention to support CUDA Graph?

Open
#3,002 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

vLLM v1 uses CUDA Graph to capture the execution workflow of the entire model, resulting in significant performance improvements compared to the previous version. I'm wondering if there are any plans to support CUDA Graph in Candle. Would it be possible to add `start_capture`, `end_capture`, and `replay` to the `Module` so that the captured graph can be replayed within the forward method? @LaurentMazare

Eric may also be interested in this @EricLBuehler

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing Candle's Module abstraction and the forward method, which are the entry points named in the issue. Determine whether CUDA Graph capture and replay fit those interfaces, then clarify the required API and scope; the issue does not name files or tests, and completion is not yet defined.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.