microsoft / microsoft/foundry-local
[Bug] Vision API VRAM Leak (onnxruntime-genai)
- Dominant language
- C++
- Stars
- 2.6k
- Forks
- 369
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 39
Description
### Describe the issue
**Vision API VRAM Leak (onnxruntime-genai)**
When running vision queries through the qwen3.5-4b model, the underlying engine (onnxruntime-genai) does not fully release VRAM between inference calls ; it climbs to ~11.9 GB over 3 requests, latency escalates from 54s→90s, then crashes with a CUDA error in CudaMallocArray...out of memory.
-_**Impact:**_ Runtimes escalated severely (from 54 seconds to 90 seconds) before crashing entirely with a CUDA error in CudaMallocArray... out of memory.
### To reproduce
Run 3+ consecutive vision queries against the vision model without unloading(qwen3.5-4b). Compare against the workaround queue in local-rag/src/vision-model-guard.js, which force-unloads/reloads every 2 requests.
### Urgency
High causes a hard crash in productio.
### Platform and architecture
Windows X64
### OS Version
Windows 11
### Installation type
Released package/binary
### Foundry Local version
0.10.2
### API or surface area
Other / Unknown
### Hardware acceleration/backend
CPU
### Backend/runtime version
_No response_
Contributor guide
Research direction
Reproduce the leak with three or more consecutive vision queries against qwen3.5-4b on Windows 11, then compare the behavior with local-rag/src/vision-model-guard.js, which force-unloads and reloads after two requests. Track VRAM, latency, and the CudaMallocArray out-of-memory failure; done requires identifying the release failure and a validated fix or clear upstream runtime diagnosis.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, javascript
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100