MCP robustness roadmap: Track A Phase 1 kickoff and inbound MCP server plan

## Summary

Create and execute the staged MCP robustness roadmap for SimpleChat, starting with Track A Phase 1 outbound MCP plugin hardening and carrying forward the enterprise-ready plan for a future inbound SimpleChat MCP server.

This issue is based on `docs/explanation/features/MCP_PLUGIN_ROBUSTNESS_PLAN.md` and incorporates review of PR #722 as prototype/context only. PR #722 should inform the initial inbound tool set and PRM/bearer-token requirements, but its external MCP server implementation should not be copied directly because it is not robust, scalable, or enterprise-ready enough for production.

Related context:

- Planning doc: `docs/explanation/features/MCP_PLUGIN_ROBUSTNESS_PLAN.md`
- Prototype/reference PR: #722
- Related older MCP workflow issue: #116

## User Value

- Improves reliability for outbound MCP actions, especially Splunk/token-based `streamable_http` servers.
- Reduces secret leakage, auth ambiguity, endpoint misconfiguration, and diagnostics gaps in MCP actions.
- Establishes a safe roadmap for exposing SimpleChat as an inbound MCP server without bypassing existing governance, workspace roles, or enterprise security boundaries.
- Gives admins a clear path to govern MCP clients, tools, scopes, and future group/public workspace access.

## Plan Overview

### Track A: Outbound MCP Plugin Robustness

SimpleChat acts as an MCP client that connects to external MCP servers.

1. **Phase 1: Baseline hardening for Splunk and token-based MCP**
- Add a Splunk-friendly preset/profile.
- Add safe custom HTTP headers.
- Improve endpoint and transport validation.
- Add timeout/retry controls.
- Improve error classification and redaction.
- Add focused functional and UI tests.

2. **Phase 2: Capability probe and tool metadata robustness**
- Add compatibility probe/discovery mode.
- Preserve richer tool metadata.
- Add opt-in argument schema validation.
- Add result-size policies.
- Add UI warnings for discovery/tool issues.

3. **Phase 3: TLS and enterprise network options**
- Confirm connector support first.
- Prefer OS/container trust store guidance.
- Add certificate references only through Key Vault/workspace identity patterns if supported.

4. **Phase 4: OAuth 2.1 PKCE**
- Add explicit OAuth fields and callback flows.
- Keep external MCP OAuth token storage separate from the app sign-in MSAL cache.
- Store secrets/tokens securely and add redaction/tests.

5. **Phase 5: Advanced MCP features**
- Prompts/resources only after UX design.
- Streaming/large-result handling after result policies are stable.
- Long-running calls only with persisted jobs and cleanup/retention rules.

### Track B: Inbound SimpleChat MCP Server

SimpleChat exposes a governed MCP server surface for approved MCP clients.

1. **Phase B0: Architecture decision**
- Decide first production hosting model.
- Prefer a first-class SimpleChat component/service layer initially.
- Keep business logic behind reusable auth-aware tool services.
- Start with streamable HTTP.

2. **Phase B1: Auth foundation and PRM**
- Add a dedicated inbound MCP auth guard.
- Do not broaden the shared `accesstoken_required` external-route decorator.
- Validate token issuer, tenant, expiration, and audience.
- Require a dedicated role/scope and approved caller app ID.
- Separate app-only and delegated user access.
- Add `.well-known/oauth-protected-resource` metadata.

3. **Phase B2: Governance, roles, and tool policy**
- Use Entra roles/scopes as the coarse access gate.
- Use app ID allowlists for client trust.
- Use SimpleChat governance for client/tool/scope policy.
- Use existing workspace roles for data access.
- Deny by default when client, tool, or scope is not explicitly allowed.

4. **Phase B3: Initial read-only personal tools**
- `show_user_profile`
- `list_conversations`
- `get_conversation_messages`
- `list_personal_documents`
- `list_personal_prompts`
- `search_personal_documents`
- `list_agent_template_tags`

5. **Phase B4: Personal chat write tool**
- Add `send_personal_chat_message` only after read-only tools, auth, governance, telemetry, and tests are stable.

6. **Phase B5: Deferred group/public/all-scope tools**
- Keep group, public, and all-scope search/chat tools disabled until explicit governance and workspace-role checks are wired and tested.

7. **Phase B6: Enterprise readiness**
- Add observability, correlation IDs, structured logging, redaction, rate limits, health/readiness endpoints, stateless/multi-instance behavior, and operational documentation.

## Phase 1 Kickoff Scope

Start implementation with **Track A Phase 1** only:

- [ ] Add a Splunk-friendly MCP preset/profile.
- [ ] Add `custom_headers` support in MCP `additionalFields`.
- [ ] Validate header names using a strict allowlist.
- [ ] Redact custom header values in logs, API responses, UI summaries, and tests.
- [ ] Merge custom headers with auth headers in `McpPluginFactory._build_headers`, with auth headers winning by default.
- [ ] Improve endpoint validation for `streamable_http`, `sse`, and `websocket` transports.
- [ ] Preserve the current restriction that `stdio` is only available for admin-managed global actions.
- [ ] Add optional retry count/backoff settings for discovery and tool calls without changing existing defaults unexpectedly.
- [ ] Classify DNS/connect, TLS, authentication, MCP initialization, discovery, timeout, and tool execution failures.
- [ ] Extend MCP schema/UI and validation as needed.
- [ ] Add focused functional and UI tests.
- [ ] Bump `application/single_app/config.py` version when code changes are made.

## Acceptance Criteria

- [ ] Existing MCP action manifests remain backward compatible.
- [ ] Splunk/token-based `streamable_http` MCP setup is easier and safer to configure.
- [ ] Custom headers are validated, merged, and redacted consistently.
- [ ] Endpoint and transport validation rejects unsupported or malformed values.
- [ ] Timeout/retry configuration is bounded and preserves current default behavior.
- [ ] Discovery and tool-call failures return actionable, redacted error categories.
- [ ] Tests cover bearer header construction, custom header validation/redaction, discovery secret hydration, invalid/expired token error shapes, tool allowlist behavior, and manifest normalization.
- [ ] UI tests cover the updated MCP action modal behavior.
- [ ] Route policy tests are run if route files change.
- [ ] No secrets appear in logs, rendered HTML, browser-visible JSON, docs examples, or test artifacts.
- [ ] Inbound MCP work is not started until the Track B auth/governance architecture is explicitly reviewed.

## Risks And Guardrails

- Do not relax shared external auth decorators to accept broad `User` or `Admin` roles for inbound MCP.
- Do not copy PR #722's external MCP server directly into production.
- Do not rely on process-local token/session caches for production auth state.
- Do not expose group, public, or all-scope inbound MCP tools in the first inbound slice.
- Do not ship user-data inbound tools until token audience/issuer validation, client allowlisting, governance, and delegated-user isolation are complete.
- Keep all browser JavaScript/CSS assets local and CSP-compatible if UI changes are needed.

## Suggested Labels / Triage

- Priority: P1
- Size: XL
- Labels: `enhancement`, `security_improvement`

## Implementation Issue Breakdown

Use this issue as the parent/epic. The implementation work is split into the following focused issues so each slice can be reviewed, implemented, and validated independently:

- [ ] #1014 - Track A Phase 1: harden outbound token-based MCP actions
- [ ] #1015 - Track A Phase 2: add capability probe, richer metadata, and result policies
- [ ] #1016 - Track A Phase 3-4: enterprise TLS options and outbound OAuth PKCE
- [ ] #1017 - Track B Phase B0-B2: design inbound MCP architecture, auth, PRM, and governance
- [ ] #1018 - Track B Phase B3: expose initial read-only personal SimpleChat MCP tools
- [ ] #1019 - Track B Phase B4: add guarded personal chat write tool
- [ ] #1020 - Track B Phase B5-B6: enable governed group/public tools and enterprise readiness

## Recommended Execution Order For Today

1. Start with #1014 to get Track A Phase 1 outbound MCP hardening landed.
2. Move to #1017 before any inbound tools are exposed, because inbound MCP requires the auth/governance foundation first.
3. Implement #1018 after #1017 is stable.
4. Implement #1019 after read-only personal tools are validated.
5. Continue with #1015 and #1016 for richer outbound MCP functionality.
6. Finish with #1020 only after group/public/all-scope governance and workspace-role checks are explicitly tested.

## Project Metadata Target

- Priority: P1
- Parent size: XL
- Work item sizes:
- #1014: L
- #1015: M
- #1016: L
- #1017: XL
- #1018: L
- #1019: L
- #1020: XL