microsoft / microsoft/foundry-local

Expose ONNX Runtime memory management options (arena settings, GPU memory limits) through the CLI and SDK [SDK] & [CLI]

Open
#861 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
2.6k
Forks
369
Avg merge
2d 17h
Merged PRs (30d)
39

Description

I'm running Foundry Local as part of a local-first RAG application (Python SDK, also tested via CLI). Over extended runtime, the loaded model's memory usage grows and eventually causes inference failures — consistent with known ONNX Runtime memory arena behavior (e.g. enable_cpu_mem_arena pre-allocation, GPU arena growth not being fully released between runs; see microsoft/onnxruntime#11627 and microsoft/onnxruntime#19445).
Currently, neither the CLI nor the SDK exposes any way to tune this. The only available mitigation is:

SDK: calling model.unload() followed by model.load() to fully reset the session
CLI: attempting foundry service restart, which in my case fails outright and isn't a reliable substitute

Both are blunt workarounds that discard the whole session/service rather than tuning the underlying memory behavior.

SDK: Expose a way to pass ONNX Runtime SessionOptions and execution-provider options (e.g. enable_cpu_mem_arena, enable_mem_pattern, gpu_mem_limit, arena_extend_strategy) through the Configuration object or model.load().Even a minimal passthrough — e.g. Configuration(session_options_overrides={...}) — would help.

CLI: Expose equivalent flags on foundry model load (and/or foundry service start)

CLI: Expose equivalent flags on foundry model load (and/or foundry service start

Manually unloading/reloading the model (SDK) or restarting the service (CLI) on a request counter or timer — functional, but doesn't address root cause, adds latency spikes on recycle, and in our case foundry service restart fails and isn't usable at all.
Bypassing Foundry Local entirely and loading the ONNX model directly via onnxruntime.InferenceSession — loses Foundry Local's hardware auto-detection, execution provider management, and lifecycle handling, which is the main reason we adopted it in the first place.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the SDK Configuration and model.load() paths and the CLI foundry model load entry point, then inspect how runtime and execution-provider options are currently constructed. Compare those paths with the requested ONNX Runtime session and GPU memory settings. Done means supported options can be configured through both SDK and CLI without requiring model reloads or service restarts.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
ai, api, cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.