microsoft / microsoft/foundry-local
Expose ONNX Runtime memory management options (arena settings, GPU memory limits) through the CLI and SDK [SDK] & [CLI]
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 2.6k
- Forks
- 369
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 39
Description
I'm running Foundry Local as part of a local-first RAG application (Python SDK, also tested via CLI). Over extended runtime, the loaded model's memory usage grows and eventually causes inference failures — consistent with known ONNX Runtime memory arena behavior (e.g. enable_cpu_mem_arena pre-allocation, GPU arena growth not being fully released between runs; see microsoft/onnxruntime#11627 and microsoft/onnxruntime#19445).
Currently, neither the CLI nor the SDK exposes any way to tune this. The only available mitigation is:
SDK: calling model.unload() followed by model.load() to fully reset the session
CLI: attempting foundry service restart, which in my case fails outright and isn't a reliable substitute
Both are blunt workarounds that discard the whole session/service rather than tuning the underlying memory behavior.
SDK: Expose a way to pass ONNX Runtime SessionOptions and execution-provider options (e.g. enable_cpu_mem_arena, enable_mem_pattern, gpu_mem_limit, arena_extend_strategy) through the Configuration object or model.load().Even a minimal passthrough — e.g. Configuration(session_options_overrides={...}) — would help.
CLI: Expose equivalent flags on foundry model load (and/or foundry service start)
CLI: Expose equivalent flags on foundry model load (and/or foundry service start
Manually unloading/reloading the model (SDK) or restarting the service (CLI) on a request counter or timer — functional, but doesn't address root cause, adds latency spikes on recycle, and in our case foundry service restart fails and isn't usable at all.
Bypassing Foundry Local entirely and loading the ONNX model directly via onnxruntime.InferenceSession — loses Foundry Local's hardware auto-detection, execution provider management, and lifecycle handling, which is the main reason we adopted it in the first place.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the SDK Configuration and model.load() paths and the CLI foundry model load entry point, then inspect how runtime and execution-provider options are currently constructed. Compare those paths with the requested ONNX Runtime session and GPU memory settings. Done means supported options can be configured through both SDK and CLI without requiring model reloads or service restarts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- ai, api, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100