microsoft / microsoft/agent-framework

Python: perf(foundry-hosting): FoundryAgentSessionStore rebuilds FoundryStateStore (credential + metadata round-trip) on every request

Open
#8,280 1 comment 0 reactions 1 assignee View on GitHub

@eavanvalkenburg is already working on this.

Since Sep 14, 2026.

foundry hosting python reproduced
Dominant language
Python
Stars
13.6k
Forks
2.3k
Avg merge
2d 45m
Merged PRs (30d)
358

Description

### Describe the bug / perf issue

`FoundryAgentSessionStore` (the default agent-session `SessionStore` for hosted MAF agents) rebuilds its backing `FoundryStateStore` on **every** `get`/`set`/`delete`. Each `_get_store()` call goes through `FoundryStateStore.get_or_create("agent_sessions", user_isolation=True)`, which:

1. constructs a fresh credential (empty token cache → a new managed-identity token fetch), and
2. issues an `agent_sessions` **metadata round-trip** (`GET`/`POST state_stores`) *before* the actual item operation.

Because the hosting infra calls `set()` in the `finally` of every Responses request, this redundant credential + metadata work lands on the **critical path of every request**.

### Component

`python` / `foundry_hosting` — `FoundryAgentSessionStore` (`_state_store.py`).

### Expected behavior

The agent-session scope (`"agent_sessions"`, `user_isolation=True`) is identical for every request, so the backing store should be resolved once and reused for the process lifetime. Only the real item `GET`/`PUT`/`DELETE` should remain on the hot path — not a per-request credential build + metadata round-trip.

### Measured impact

500 cold + 500 warm streaming requests per agent, concurrency 50, private/VNet Foundry project, `gpt-4o-mini`, 0 errors / 0 throttled. Same-conditions A/B, baseline vs cached Responses agent (ms, p50):

| Metric | Baseline | Cached | Δ |
|---|---|---|---|
| WARM TTLB p50 | 7,736 | 7,464 | **−272 ms (−3.5%)** |
| WARM TTFB p50 | 5,680 | 5,683 | ~flat |

The win is the elimination of the per-request credential + metadata round-trip; the remaining warm latency is the model first-token floor (~5.7 s TTFB), unaffected by this change.

### Proposed fix

Cache one **per-event-loop, per-scope** `FoundryStateStore` for the agent-session scope (guarded by a per-loop lock with double-checked init; keyed by the running loop via a `WeakKeyDictionary` so a closed loop's store is GC'd, and keyed by scope so subclasses overriding `DEFAULT_ROOT_SCOPE` are isolated). Item `get`/`set`/`delete` semantics stay byte-for-byte identical; checkpoint / function-approval stores are left untouched (not on the per-request hot path).

Fix implemented in PR #8178 (closed pending this tracking issue).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.