microsoft / microsoft/foundry-local
[Bug] Model.UnloadAsync retains process and CUDA memory after model unload
- Dominant language
- C++
- Stars
- 2.6k
- Forks
- 369
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 39
Description
### Describe the issue
After loading `qwen3.5-2b-text-cuda-gpu:1`, attempting inference, and calling `IModel.UnloadAsync()`, most model-related process and CUDA device memory remains allocated.
`IsLoadedAsync()` returns `false`, but measured memory remains close to the peak allocation. The expected behavior is for unloading to release memory that is no longer needed.
In a reproducible run, inference failed with a CUDA allocation error after the model loaded. Unloading afterward retained approximately 98% of the peak process-private allocation and approximately 99% of the peak CUDA allocation. Exact values vary; the issue is the retained proportion after unload.
### To reproduce
1. Use `Microsoft.AI.Foundry.Local` version `2.0.1`.
2. Initialize the manager and register `CUDAExecutionProvider`.
3. Download and select `qwen3.5-2b-text-cuda-gpu:1`.
4. Record process-private and CUDA device memory.
5. Call `LoadAsync()` and attempt a chat completion.
6. Call `UnloadAsync()`, then force managed collection and finalization.
7. Confirm `IsLoadedAsync()` returns `false` and measure memory again.
Observed behavior:
```text
MODEL_LOADED_AFTER_UNLOAD=False
RETENTION kind=PROCESS_PRIVATE retained_percent=approximately 98
RETENTION kind=GPU_DEVICE_USED retained_percent=approximately 99
```
### Urgency
Retained allocations prevent applications from reliably recovering memory after unloading a model, especially after an inference allocation failure.
### System information
- Platform and architecture: Windows X64
- OS version: Windows 11
- Installation type: Released package/binary
- Foundry Local version: Microsoft.AI.Foundry.Local 2.0.1
- API or surface area: C# SDK
- Hardware acceleration/backend: WebGPU/CUDA
- Backend/runtime version: ONNX Runtime 1.28.0; ONNX Runtime GenAI 0.15.2
Contributor guide
Assessment
This issue has not been assessed yet.