mcp: Implement Automatic OAuth Discovery via 401 Detection

Scope

Implement automatic OAuth discovery when MCP servers return 401 Unauthorized responses, following the MCP spec's discovery mechanisms. This enables zero-configuration OAuth for compliant servers.

Haystack MCP Integration Details

401 Detection Points:

• StreamableHttpClient.connect(): Catch 401 during initial connection/session establishment
• StreamableHttpClient.call_tool(): Catch 401 during tool invocations
• Both should trigger the same discovery and OAuth flow

Discovery Flow:

1. Parse WWW-Authenticate: Bearer ... header to extract metadata hints
2. Try WWW-Authenticate header resource_metadata parameter first
3. Fallback to /.well-known/oauth-protected-resource{path} (with path insertion per RFC 9728)
4. Fallback to /.well-known/oauth-protected-resource (root)
5. Extract authorization_servers list from resource metadata
6. Fetch authorization server metadata from /.well-known/oauth-authorization-server
7. Extract authorization_endpoint, token_endpoint, registration_endpoint, etc.
8. Validate S256 PKCE support (required by MCP spec)

Dynamic Client Registration (DCR):

• Check for registration_endpoint in AS metadata
• POST to register client and obtain client_id (and client_secret if confidential)
• Fallback to requiring user-provided client_id if DCR not supported
• This enables truly zero-config OAuth for servers that support it

Auto-Population of OAuthConfig:

• Create OAuthConfig instance from discovered metadata
• Fill in authorization_endpoint, token_endpoint from metadata
• If DCR succeeds, populate client_id and client_secret
• Store discovered config for reuse (avoid repeated discovery)

Retry Logic:

• After obtaining token, retry the original request that triggered 401
• Cache token in server_info.headers["Authorization"] for subsequent requests
• Handle token expiration and refresh (if refresh token available)

Integration with Existing Code:

• Discovery should be non-invasive - existing code should work without OAuth
• If OAuthConfig is provided, use it (still perform discovery for endpoints if not specified)
• Discovery should happen lazily (only on 401) to avoid unnecessary requests
• Consider caching discovered metadata to avoid repeated discovery

Error Handling:

• If discovery fails, provide clear error messages
• If DCR fails, fallback to requiring user-provided credentials
• If OAuth flow fails, propagate meaningful errors

Requirements

• Catch 401 Unauthorized responses in StreamableHttpClient.connect() and call_tool()
• Parse WWW-Authenticate: Bearer ... header to extract metadata hints
• Implement automatic resource metadata discovery:
  • Try WWW-Authenticate header resource_metadata parameter first
  • Fallback to /.well-known/oauth-protected-resource{path} (with path insertion)
  • Fallback to /.well-known/oauth-protected-resource (root)
• Implement automatic authorization server metadata discovery:
  • Fetch /.well-known/oauth-authorization-server from discovered issuer
  • Extract authorization_endpoint, token_endpoint, code_challenge_methods_supported
  • Validate S256 PKCE support (required by MCP spec)
• (Optional) Implement Dynamic Client Registration (RFC 7591):
  • Check for registration_endpoint in AS metadata
  • POST to register client and obtain client_id (and client_secret if confidential)
  • Fallback to requiring user-provided client_id if DCR not supported
• Auto-populate OAuthConfig from discovered metadata
• Trigger appropriate OAuth flow (Authorization Code or Client Credentials) based on discovery
• Retry original request after obtaining token
• Cache discovered metadata and tokens to avoid repeated discovery
• Handle errors gracefully with clear messages
• Add comprehensive tests for discovery flow
• Add tests for 401 handling and retry logic
• Add tests for DCR (if implemented)

References

• MCP Authorization Spec - Discovery
• RFC 9728 - OAuth 2.0 Protected Resource Metadata
• RFC 8414 - OAuth 2.0 Authorization Server Metadata
• RFC 7591 - OAuth 2.0 Dynamic Client Registration

Start with StreamableHttpClient.connect() and call_tool(), tracing how 401 responses and OAuthConfig are currently handled. Review the MCP discovery, RFC 9728, RFC 8414, and RFC 7591 references, then add focused tests for metadata discovery, retry behavior, caching, and errors. Done means both entry points share lazy discovery and can retry authenticated requests without breaking existing OAuth configuration.