MCPRemoteProxy + embeddedAuthServer: upstreamswap does not inject the stored per-user upstream token; the AS's own JWT is forwarded to the remote MCP (v0.49.0)

Summary

On MCPRemoteProxy with an MCPExternalAuthConfig of type embeddedAuthServer, the
per-user OAuth flow completes correctly and the upstream token is persisted in Redis,
but the upstreamswap middleware never injects it. The proxy forwards the JWT issued by
its own embedded authorization server to the remote MCP server, which rejects it because
the audience is the proxy, not the remote.

Net effect: login works end to end, the audit log shows the correct user, and the MCP
returns zero tools for every client.

Environment

• ToolHive proxyrunner v0.49.0 (ghcr.io/stacklok/toolhive/proxyrunner:v0.49.0)
• Kubernetes operator, CRDs MCPRemoteProxy + MCPExternalAuthConfig + MCPOIDCConfig
• Remote MCP: a third-party hosted MCP that implements the MCP authorization spec
  (RFC 9728 metadata, its own authorization server)
• Auth server storage: Redis
• Client: mcp-remote

Configuration (trimmed)

kind: MCPRemoteProxy
spec:
  remoteUrl: https://api.vendor.example.com/api/2.0/mcp/mcp
  transport: streamable-http
  authServerRef:
    kind: MCPExternalAuthConfig
    name: vendor-embedded-auth
  oidcConfigRef:
    name: vendor-embedded-oidc
    audience: "https://mcp-proxy.internal.example.com/mcp"
    resourceUrl: "https://mcp-proxy.internal.example.com/mcp"
    scopes: [openid, email, profile, offline_access]
  sessionStorage:
    provider: redis
  trustProxyHeaders: true
---
kind: MCPExternalAuthConfig
spec:
  type: embeddedAuthServer
  embeddedAuthServer:
    issuer: "https://mcp-proxy.internal.example.com"
    storage:
      type: redis
    upstreamProviders:
      - name: vendor
        type: oidc
        oidcConfig:
          issuerUrl: https://oauth.vendor.example.com
          redirectUri: "https://mcp-proxy.internal.example.com/oauth/callback"
          scopes: [openid, profile, email, offline_access]
          dcrConfig:
            discoveryUrl: https://oauth.vendor.example.com/.well-known/oauth-authorization-server
---
kind: MCPOIDCConfig
spec:
  type: inline
  inline:
    issuer: "https://mcp-proxy.internal.example.com"
    jwksAllowPrivateIP: true

What happens

The remote MCP returns:

HTTP/2 401
www-authenticate: Bearer error="invalid_token"
  error_description="OAuth token claim validation failed: jwt audience invalid"
  resource_metadata="https://api.vendor.example.com/api/2.0/auth/.well-known/oauth-protected-resource"

{"jsonrpc":"2.0","error":{"code":-32001,
 "message":"OAuth token claim validation failed: jwt audience invalid"},"id":null}

Decoding the token the proxy sent upstream shows it is the embedded AS's own token:

{
  "iss": "https://mcp-proxy.internal.example.com",
  "aud": ["https://mcp-proxy.internal.example.com/mcp"],
  "tsid": "SWBDJ5GQPUFOIWIPKPRBOHKLOX",
  "scp": ["openid","email","profile","offline_access"]
}

Proxy log:

WARN received 401 Unauthorized response for remote server, marking as unauthenticated server=vendor

The middleware is configured, and the token it needs is present

upstreamswap is in the chain, from the generated runconfig.json:

{"type": "bodylimit",     "parameters": {"max_bytes": 8388608}}
{"type": "audit",         "parameters": {...}}
{"type": "auth",          "parameters": {"oidc_config": {...},
                           "embedded_auth_server_issuer": "https://mcp-proxy.internal.example.com"}}
{"type": "upstreamswap",  "parameters": {"config": {"provider_name": "vendor"}}}
{"type": "mcp-parser",    "parameters": {}}
{"type": "usagemetrics",  "parameters": {}}
{"type": "recovery",      "parameters": null}

And the per-user upstream token is persisted, correctly indexed:

thv:auth:{data:vendor}:upstream:SWBDJ5GQPUFOIWIPKPRBOHKLOX:vendor
thv:auth:{data:vendor}:upstream:idx:SWBDJ5GQPUFOIWIPKPRBOHKLOX
thv:auth:{data:vendor}:user:upstream:<redacted>
thv:auth:{data:vendor}:user:providers:<redacted>
thv:auth:{data:vendor}:provider:10:vendor:<redacted>
thv:auth:{data:vendor}:dcr:<redacted>

The key point: SWBDJ5GQPUFOIWIPKPRBOHKLOX in those Redis keys is exactly the tsid
claim carried by the JWT the client presents on every request (see the decoded token above).

So the middleware receives, inside the very token it just validated, the exact key under
which that user's upstream token is stored — and still forwards the internal JWT.

Expected vs actual

Expected: upstreamswap replaces the Authorization header with the upstream access
token for the provider named vendor, resolved from the authenticated session.

Actual: the embedded AS's own JWT is forwarded unchanged to remoteUrl.

What the client sees

mcp-remote never surfaces the 401. It aborts earlier on SEP-2352 resource validation,
because the upstream's resource_metadata names a different resource than the proxy:

Protected resource https://api.vendor.example.com/api/2.0/mcp/mcp
  does not match expected https://mcp-proxy.internal.example.com/mcp (or origin)

The end user just sees a connection timeout, which makes this look like a network problem
rather than an auth one.

Ruled out

- Upstream OAuth leg is wired and working. /oauth/authorize 302s to the vendor's
  authorization endpoint with a DCR-registered client_id, correct redirect_uri, PKCE
  S256 and offline_access. The user completes the vendor login.
- Identity resolves. The audit log records the correct user and user id.
- Not a missing token. See the Redis keys above.
- Not the OIDC-discovery fallback. We separately hit oidc discovery failed ... the provided URL redirects to a private IP address because our issuer hostname resolves to
  an internal load balancer. Setting jwksAllowPrivateIP: true removed those errors
  entirely — and the 401 persists unchanged. Two independent problems.
- Not #6395. That is about unattended/machine clients with no interactive login. Here
  the client is interactive and has completed the browser flow.
- Not #5383. That is about composing per-user backends behind a vMCP. This is a
  standalone MCPRemoteProxy with no groupRef, which #5383 names as the pattern that
  should work today.

Reproduction

1. Deploy an MCPRemoteProxy against any remote MCP that validates the aud claim, with
   an embeddedAuthServer whose upstream is that vendor's OAuth.
2. Complete the interactive login from an MCP client.
3. Send initialize. The remote rejects with an audience/invalid-token 401.
4. Confirm in Redis that …:upstream:<tsid>:<provider> exists and that <tsid> matches
   the tsid claim of the client's token.

Note

There is no upgrade path from our side — v0.49.0 is the latest release at the time of
writing.

Start at the upstreamswap middleware and its provider_name vendor configuration, then trace how the authenticated token's tsid is used to look up the Redis-stored upstream token. Reproduce the interactive login and initialize request, and verify that the remote receives the stored vendor token rather than the embedded authorization server JWT and no longer returns the audience-related 401.