Egress OBO: optional caching of exchanged tokens in the configured SecretStore

## Summary

Add an optional cache for Microsoft Entra OBO (`obo_exchange`) tokens so deployments can choose to reuse a still-valid exchanged token rather than performing a new token exchange against Entra for every MCP request.

The existing behavior should remain available and should preferably remain the default.

## Current behavior

The current OBO implementation is deliberately stateless.

`auth_server/egress_obo.py` documents the following security invariant:

```python
# The minted token embeds the user's ``sub``; it is exchanged PER REQUEST and is
# NEVER cached or reused across users. This module holds no cache.
```

Each request to an MCP server configured with:

```text
egress_auth_mode=obo_exchange
```

therefore performs an OBO token exchange against the configured IdP before forwarding the request to the MCP server.

For Entra this means that, conceptually:

```text
MCP request
|
v
Gateway
|
+----> Entra token endpoint
| OBO exchange
|<---- access token
|
+----> MCP server
```

This has a strong security property: the gateway does not retain the resulting OBO token and every request receives a freshly exchanged token.

It also means that every MCP invocation incurs an additional network request to Entra.

## Motivation

Some deployments may prefer to trade a small amount of the current security/isolation property for reduced token-exchange overhead.

For workloads involving frequent MCP calls, performing an Entra OBO exchange for every individual MCP request can introduce:

* additional request latency;
* additional dependency on Entra availability on the MCP request hot path;
* increased calls to the Entra token endpoint;
* additional operational load and possible throttling considerations.

The gateway already has a pluggable per-user credential `SecretStore` supporting AWS Secrets Manager and OpenBao.

It would be useful to optionally cache short-lived OBO access tokens there and reuse them until they approach expiry.

This assumes that, for a given deployment, reading a cached credential from the configured SecretStore is operationally preferable to performing another OBO exchange with Entra.

That may not be true for every deployment, which is why this behavior should be optional.

## Historical context

The original OBO feature request (#1269) already anticipated this possibility:

> An exchanged token may be briefly cached, default off, capped well below its expiry.

The implementation shipped in #1420 chose the stronger stateless model and explicitly introduced a per-request/no-cache invariant.

This request proposes implementing the optional caching mode originally contemplated by #1269 without changing the existing stateless mode.

## Proposed behavior

Introduce an optional OBO token cache.

For example:

```text
EGRESS_OBO_CACHE_ENABLED=false
```

When disabled:

```text
MCP request
|
v
OBO exchange with Entra
|
v
MCP server
```

This is the current behavior.

When enabled:

```text
MCP request
|
v
Look up cached OBO token
|
+-- valid token found ------> MCP server
|
+-- missing / expiring
|
v
Entra OBO exchange
|
v
cache token
|
v
MCP server
```

The default should remain `false`, preserving today's security model and behavior.

## Cache backend

The cache should preferably reuse the existing configured `SecretStore`:

```text
SECRET_STORE_BACKEND=secrets-manager
```

or:

```text
SECRET_STORE_BACKEND=openbao
```

rather than introducing a separate persistence system specifically for OBO.

For AWS deployments, cached OBO tokens could therefore live in the same per-principal Secrets Manager structure already used for egress credential material, using a separate OBO cache namespace/key.

For OpenBao, they could live under the same per-principal hierarchy using a dedicated OBO cache path.

The OBO cache entry must be distinguishable from long-lived PAT and `oauth_user`/3LO credential entries.

## Cache identity

A cached OBO token must never be keyed only by user.

At minimum the cache identity should include:

```text
canonical auth_method
canonical user_id
target_audience
effective scopes
```

and, where relevant, IdP/tenant identity.

Conceptually:

```text
OBO cache key =
(
auth_method,
user_id,
target_audience,
normalized_scopes
)
```

A hash of normalized scopes may be used in the physical storage key.

This ensures that a token obtained for one audience or delegated scope set cannot accidentally be reused for another.

If other properties of the ingress assertion can materially alter the authorization represented by the exchanged token, those should also be considered when defining the safe cache key.

## Expiration

Cached OBO tokens must always have bounded lifetime.

The gateway should derive the real token expiration from the Entra response and/or validated JWT `exp`, and never reuse an expired or near-expiry token.

A configurable maximum cache lifetime could additionally cap reuse:

```text
EGRESS_OBO_CACHE_MAX_TTL_SECONDS=300
```

The effective cache lifetime should be:

```text
min(
token_remaining_lifetime - safety_skew,
EGRESS_OBO_CACHE_MAX_TTL_SECONDS
)
```

For example, an Entra token valid for another 55 minutes could still be intentionally cached for only 5 minutes.

This allows operators to choose how much token reuse they are comfortable with independently of the lifetime Entra gives the access token.

The gateway should never extend the lifetime of an OBO token.

## No refresh-token semantics

OBO caching should remain fundamentally different from the `oauth_user` 3LO vault.

A cached OBO token is just a short-lived optimization.

When it expires or is no longer eligible for reuse, the gateway should perform another OBO exchange using the current ingress user token.

There should be no requirement to obtain or retain an OBO refresh token.

## Failure behavior

Caching must never weaken the existing fail-closed behavior.

In particular:

* A cache lookup failure must not cause an expired token to be reused.
* An expired or near-expiry cached token must result in a fresh OBO exchange.
* An OBO exchange failure must not cause the gateway to fall back to a stale cached token outside its permitted cache lifetime.
* A cached token for another user, audience, tenant, or scope set must never be reused.
* Failure to write the newly exchanged token to the cache should preferably still allow the current request to proceed with the freshly exchanged token; the cache is an optimization, not the authority granting access.

## Concurrency / token-exchange storms

In a multi-replica deployment, multiple requests for the same user/audience may arrive simultaneously immediately after the cached token expires.

The implementation should avoid turning one cache miss into many simultaneous Entra exchanges where practical.

A per-cache-key single-flight / short distributed lease could ensure that one request performs the exchange while concurrent requests reuse the resulting token.

This is especially useful because reducing Entra token-endpoint traffic is one of the main motivations for the feature.

## Security trade-off

This option intentionally introduces a security/performance trade-off.

### Cache disabled

Advantages:

* no OBO token persisted by the gateway;
* fresh OBO exchange for every request;
* minimum token-reuse window;
* current security model is preserved.

Cost:

* every MCP call requires a roundtrip to Entra.

### Cache enabled

Advantages:

* fewer Entra token exchanges;
* potentially lower MCP request latency;
* reduced dependency on Entra latency for cache-hit requests;
* reduced token-endpoint traffic.

Cost:

* an OBO bearer token exists in the gateway's credential store for a bounded period;
* revocation / authorization changes may take effect less immediately than with a fresh exchange on every request;
* compromise of the credential store introduces an additional short-lived bearer-token exposure.

This trade-off should be clearly documented so operators can make an explicit decision based on their deployment's threat model and performance requirements.

## Interaction with credential encryption

Cached OBO tokens should receive the same protections as other sensitive egress credential material.

If application-layer encryption of egress credentials is enabled/implemented, cached OBO tokens must also be encrypted before persistence to AWS Secrets Manager or OpenBao.

Caching OBO tokens must not create a weaker storage path than PAT or 3LO credentials.

If possible, it should be stored in the same secrets as the PAT and 3LO credentials.

## Observability

It would be useful to expose bounded metrics such as:

```text
obo_cache_hit
obo_cache_miss
obo_cache_expired
obo_exchange_performed
obo_exchange_failure
```

This allows operators to determine whether enabling the cache is actually reducing Entra calls and improving latency.

No token material or cache keys containing raw sensitive identity information should be logged.

## Suggested configuration

For example:

```text
EGRESS_OBO_CACHE_ENABLED=false
EGRESS_OBO_CACHE_MAX_TTL_SECONDS=300
EGRESS_OBO_CACHE_EXPIRY_SKEW_SECONDS=30
```

Exact names are implementation-defined.

The important properties are:

* cache is explicitly opt-in;
* cache reuse is bounded independently of the token's full lifetime;
* the current stateless behavior remains available.

## Acceptance criteria

* OBO caching is optional and disabled by default.
* With caching disabled, behavior remains identical to the current per-request OBO exchange.
* With caching enabled, a valid cached token can be reused without calling Entra.
* Cache entries use the configured egress `SecretStore` backend (AWS Secrets Manager or OpenBao).
* Cached OBO credentials are logically separated from PAT and 3LO entries.
* Cache identity includes at least canonical user identity, target audience and effective scopes.
* Tokens are never reused across users.
* Tokens are never reused across target audiences.
* Tokens are never reused across incompatible scope sets.
* Cached tokens are never used after their actual expiry.
* A configurable maximum cache TTL can cap reuse substantially below the token's actual expiry.
* A safety skew prevents forwarding tokens that are close to expiration.
* Cache expiry causes a new OBO exchange rather than refresh-token behavior.
* Cache write failure does not require failing a request that already has a successfully exchanged, valid token.
* Cache read errors never result in stale-token reuse.
* Exchange failures never silently fall back to expired/out-of-policy cached credentials.
* Multi-replica behavior avoids unnecessary token-exchange storms where practical.
* Cache hit/miss/exchange metrics are available without logging token material.
* Cached OBO tokens receive the same at-rest/application-layer protections as other sensitive egress credentials.
* Documentation clearly explains the security/performance trade-off.

## Why this should remain optional

The current no-cache design is a valuable security property and should not be removed.

Different deployments have different priorities:

```text
Maximum token freshness / minimal stored credential state
^
|
caching disabled
|
|
caching enabled
|
v
Lower Entra traffic / potentially lower request latency
```

Providing both modes lets operators make that trade-off explicitly rather than imposing one policy on every deployment.