microsoft / microsoft/onnxruntime

cuda graph bug for recurrent state inplace update

Open
#32,243 0 comments 0 reactions 0 assignees View on GitHub
ep:CUDA
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

Current cuda graph capture will run inference session multiple times for the inference run, like 2 times before capture to meet minimum runs requirement, then 1 time capture. If we bind input and output state to same buffer, recurrent state for LinearAttention will be wrong due to 2 extra runs before capture.

A walkaround is to make a copy of input state before graph capture, then run a graph reply with same input after graph capture is done like in https://github.com/microsoft/onnxruntime-genai/pull/2452.

We can probably do this in ORT to keep a copy of inputs (if its address is shared with some output).

Contributor guide

Open the contributing guide

Research direction

Start by tracing ONNX Runtime's CUDA graph capture path and how it handles inputs whose addresses are shared with outputs. Compare the proposed input-copy and post-capture graph replay workaround with onnxruntime-genai pull request 2452. Done means recurrent state remains correct when input and output state use the same buffer during capture, with coverage for this case.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.