stacklok / stacklok/toolhive

[vMCP] Handle auth credential expiration with retry and proactive refresh

Open
#3,869 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

authentication enhancement go vmcp
Dominant language
Go
Stars
2.2k
Forks
300
Avg merge
1d 15h
Merged PRs (30d)
184

Description

Depends on #3865

With session-scoped clients, short-lived outgoing credentials (e.g., OAuth tokens) resolved at client creation can become stale mid-session. Implement automatic client recreation on auth failure with proper safeguards.

Phase 1 (this issue):

  • Detect 401/403 from backend; recreate client with fresh credentials from OutgoingAuthRegistry
  • Retry with exponential backoff, max 3 attempts
  • Circuit-breaker: after N consecutive auth failures per backend, disable retries and log a warning
  • Use singleflight to deduplicate concurrent recreation attempts for the same backend
  • Auth-failure retries add non-trivial latency — surface this overhead in distributed traces

Phase 2 (follow-up, separate issue):

  • Modify identityPropagatingRoundTripper to read identity from request context per-operation, eliminating the stale credential problem entirely
  • For credentials with a known expiry (JWT exp, OAuth expires_in): schedule proactive client recreation before expiry
Acceptance Criteria
  • A 401 or 403 from a backend triggers client recreation with fresh credentials
  • Recreation is retried with exponential backoff up to a maximum of 3 attempts
  • After N consecutive auth failures for a backend, the circuit breaker opens and further retries are skipped
  • Concurrent auth failures for the same backend result in only one recreation attempt (via singleflight)
  • Auth-failure retry latency is visible in distributed traces
  • The raw credentials are never logged
  • Unit tests cover: successful recreation, max retries exhausted, circuit breaker opens, concurrent failures deduplicated

RFC: THV-0038 — Session-scoped client lifecycle

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with RFC THV-0038 and the session-scoped client lifecycle described here, then trace how OutgoingAuthRegistry supplies credentials and how identityPropagatingRoundTripper is used. The Phase 1 work is complete when auth failures recreate clients safely, retry with backoff and circuit-breaking, deduplicate concurrent recreation, expose retry latency in traces, avoid logging raw credentials, and pass the listed unit-test scenarios.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.