microsoft / microsoft/onnxruntime
cuda graph bug for recurrent state inplace update
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
Current cuda graph capture will run inference session multiple times for the inference run, like 2 times before capture to meet minimum runs requirement, then 1 time capture. If we bind input and output state to same buffer, recurrent state for LinearAttention will be wrong due to 2 extra runs before capture.
A walkaround is to make a copy of input state before graph capture, then run a graph reply with same input after graph capture is done like in https://github.com/microsoft/onnxruntime-genai/pull/2452.
We can probably do this in ORT to keep a copy of inputs (if its address is shared with some output).
Contributor guide
Research direction
Start by tracing ONNX Runtime's CUDA graph capture path and how it handles inputs whose addresses are shared with outputs. Compare the proposed input-copy and post-capture graph replay workaround with onnxruntime-genai pull request 2452. Done means recurrent state remains correct when input and output state use the same buffer during capture, with coverage for this case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100