microsoft / microsoft/foundry-local

[Bug] Vision API VRAM Leak (onnxruntime-genai)

Open
#1,034 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
2.6k
Forks
369
Avg merge
2d 17h
Merged PRs (30d)
39

Description

### Describe the issue

**Vision API VRAM Leak (onnxruntime-genai)**

When running vision queries through the qwen3.5-4b model, the underlying engine (onnxruntime-genai) does not fully release VRAM between inference calls ; it climbs to ~11.9 GB over 3 requests, latency escalates from 54s→90s, then crashes with a CUDA error in CudaMallocArray...out of memory.

-_**Impact:**_ Runtimes escalated severely (from 54 seconds to 90 seconds) before crashing entirely with a CUDA error in CudaMallocArray... out of memory.

### To reproduce

Run 3+ consecutive vision queries against the vision model without unloading(qwen3.5-4b). Compare against the workaround queue in local-rag/src/vision-model-guard.js, which force-unloads/reloads every 2 requests.

### Urgency

High causes a hard crash in productio.

### Platform and architecture

Windows X64

### OS Version

Windows 11

### Installation type

Released package/binary

### Foundry Local version

0.10.2

### API or surface area

Other / Unknown

### Hardware acceleration/backend

CPU

### Backend/runtime version

_No response_

Contributor guide

Open the contributing guide

Research direction

Reproduce the leak with three or more consecutive vision queries against qwen3.5-4b on Windows 11, then compare the behavior with local-rag/src/vision-model-guard.js, which force-unloads and reloads after two requests. Track VRAM, latency, and the CudaMallocArray out-of-memory failure; done requires identifying the release failure and a validated fix or clear upstream runtime diagnosis.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, javascript
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.