[PRD]: Keyless read-only MCP access via a default MCP token

Problem Statement

As an operator who wants Genie (and similar native MCP connectors) to fetch governed glossary/concept definitions from Ontos mid-answer, I can't: Genie carries one credential in the Authorization header, clears the Databricks app-gate, but is then rejected by Ontos's inner X-API-Key check. Today my only options are a token-injection shim or a proxy — both are extra moving parts to build, secure, and keep in sync with the MCP tool surface.

Background: the two gates

Ontos's MCP endpoint (/api/mcp) sits behind two independent gates:

1. App proxy (Databricks Apps OAuth2) — external, mandatory for any app endpoint. On success it injects X-Forwarded-Access-Token / X-Forwarded-Email / X-Forwarded-User. The regular HTTP API already reads these (common/authorization.py).
2. In-app MCP token (X-API-Key) — guards only /api/mcp; governs authentication, tool authorization (scopes drive tools/list filtering + tools/call enforcement via MCPHandler._has_scope), audit identity (created_by), and lifecycle (expires_at, is_active).

Native connectors like Genie clear gate 1 but present no X-API-Key, so gate 2 rejects (401 / JSON-RPC -32001). Verified end-to-end.

Established during exploration:

• MCP today never reads the forwarded identity — it's purely X-API-Key-driven — but the headers are present on every proxied request.
• The MCP token is a tool-authz gate, not a data-access gate: global_search deliberately bypasses per-user ACLs and delegates access control to token scopes.
• _has_scope is pure/data-driven, so any resolved principal's scopes flow through existing tool gating with zero tool changes.

Solution

When an MCP request arrives that has already cleared the app-gate (a real, forwarded Databricks user) but carries no X-API-Key, Ontos resolves it to an admin-designated default MCP token instead of rejecting it. That default token's scopes bound exactly what the keyless caller can do; the request is audited under the caller's forwarded email. Admins designate and scope the default token from the existing MCP-token settings UI, with a clear warning that its scopes apply to any authenticated app user. Removing/deactivating the default token turns the capability off. This lets Genie phone Ontos for governed definitions with no shim and no token dance, while keeping coarse revocation and full per-user audit attribution.

Confirmed decisions:

• Keyless caller resolves to a designated default MCP token (a real mcp_tokens row), re-stamped with the caller's forwarded email for audit.
• Default token = ordinary token row; admin sets its scopes freely (no code-enforced restriction), UI warns.
• Switch = presence: keyless works iff an active, non-expired token flagged as the keyless default exists. No separate settings flag; default off = no such row.
• Principal-resolution logic extracted onto MCPTokensManager (deep, unit-testable); routes stay thin.

User Stories

1. As a Genie user, I want Ontos glossary/concept lookups to resolve mid-answer, so that I get governed definitions without configuring an Ontos API key.
2. As an Ontos admin, I want to designate one existing MCP token as the "keyless default", so that keyless app-gate callers inherit exactly its scopes.
3. As an Ontos admin, I want to choose the default token's scopes freely in the settings UI, so that I control the keyless blast radius.
4. As an Ontos admin, I want a clear warning that the default token's scopes apply to any authenticated app user with no key, so that I don't over-grant by accident.
5. As an Ontos admin, I want keyless access disabled by default, so that upstream merges and fresh deploys never silently open this path.
6. As an Ontos admin, I want to turn keyless access off instantly by deactivating/deleting the default token, so that I retain a coarse revocation lever.
7. As an Ontos admin, I want the default token to honor expires_at, so that keyless access lapses unless deliberately re-blessed.
8. As a security reviewer, I want every keyless call attributed to the caller's forwarded email in the audit log, so that "who did what" is preserved despite a shared token identity.
9. As a security reviewer, I want keyless callers to be denied tools outside the default token's scopes (e.g. SPARQL, writes), so that least privilege holds.
10. As a security reviewer, I want a keyless request with no forwarded identity to be rejected and logged as anonymous, so that unauthenticated access is impossible.
11. As an existing keyed integration, I want my X-API-Key behavior completely unchanged, so that this feature doesn't regress me.
12. As an Ontos admin, I want at most one active keyless-default token at a time, so that the keyless capability is unambiguous.
13. As an Ontos admin, I want designating a new default to clear the previous one, so that switching defaults is a single action.
14. As an Ontos admin, I want the token list to show which token is the keyless default, so that I can see the current state at a glance.
15. As a developer, I want principal resolution in a pure, testable manager method, so that the security-relevant branches are covered by unit tests.
16. As a Genie user, I want write/mutating and SPARQL tools refused on the keyless path (assuming a read-only default), so that keyless access can't change governed state.

Implementation Decisions

• Schema: add is_keyless_default: bool (default False, indexed) to the mcp_tokens model, via an Alembic migration following the repo's short-revision convention (rebase onto development, descend from the live head — re-verify, don't hardcode).
• Repository: get_keyless_default(db) (active, non-expired, flagged row) and set_keyless_default(db, token_id) (clears any other flag then sets target) — single active default enforced in code.
• Manager (deep module): MCPTokensManager.resolve_keyless_default(email) -> Optional[MCPTokenInfo] — fetch default, update_last_used, return MCPTokenInfo built from the row but with created_by = email and a distinguishing name. Scopes taken verbatim from the row. Thread the default flag through create/designate paths.
• Routes: replace validate_api_key(...) with a thin resolve_mcp_principal(db, x_api_key, request): key present → validate_token (unchanged); key absent → read forwarded email (header-trust only, no OBO/SDK call) and resolve the default if one is active, else None. Swap the three identical call sites (SSE GET, POST handler, DELETE). Existing if not token_info: <reject/audit> blocks unchanged; keyless failures still log anonymous.
• No changes to MCPHandler, _get_username, _has_scope, session handling, or any tool — attribution and scoping flow through the existing MCPTokenInfo.
• API/admin routes: add is_keyless_default to token info responses; add an admin-only, audited action to designate an existing token as the keyless default (mirrors existing CREATE/REVOKE audit).
• Frontend: token type + create/designate calls; a "keyless default" badge/toggle in the MCP-tokens settings; an inline warning that the default token's scopes apply to any authenticated app user (call out global_search's ACL bypass); en-locale strings as source of truth.

Testing Decisions

Good tests here assert external behavior — the resolved principal and the endpoint's authz/audit outcomes — not internal call sequencing. Prior art: existing MCP token manager/route tests and the audit-logging assertions already in the suite.

• Principal resolution (unit, on the manager): key present → validates as today, default ignored; no key + active default + email → principal with created_by == email and the row's scopes; no key + no default → None; no key + inactive/expired default → None; no key + default active but no forwarded email → None.
• Single-default invariant (unit/repo): set_keyless_default clears any prior default; at most one active default exists.
• MCP route integration: with mock forwarded headers and a semantic:read+search:read default — tools/list filtered to scope; search_glossary_terms succeeds; execute_sparql_query → -32002; no email + no key → -32001 and audit username anonymous; keyless success audit username == forwarded email.

Out of Scope

• Per-caller rate limiting on the keyless path (noted as a follow-up).
• Code-enforced scope restrictions on the default token (admin choice is deliberate; UI warns instead).
• Reading/validating the OBO access token or calling the Databricks SDK on the MCP path (header-trust only, matching the regular path's trust level).
• Row-level data ACLs inside tools (global_search bypass is pre-existing behavior, unchanged).
• Multi-default or per-connector default tokens.

Further Notes

• Tradeoff accepted: keyless path loses per-token expiry/revocation granularity (revocation is coarse — one default row), but attribution is preserved via created_by = X-Forwarded-Email. Recommend enabling only where "any app user can read glossary terms" is acceptable policy.
• The default token's scopes are the entire keyless blast radius: semantic:read reaches find_entities_by_concept (lists linked data products/contracts); search:read reaches global_search (whole-index, no per-user ACL). PR should recommend read-only scopes (no *:write, sparql:query, or *).
• Chosen over the proxy/shim route because it reuses existing token machinery (UI-managed scopes, is_active revocation, expires_at) and confines the change to one resolver + three call sites.

Start with common/authorization.py, the existing MCP token manager/repository and MCP route tests to trace forwarded identity, token validation, scope filtering, and audit behavior. Implement the schema, manager, repository, route, admin API, and settings UI changes described in the issue, then run the specified principal-resolution, single-default, and MCP route integration tests. Done means keyless access is scoped, attributed, revocable, and existing keyed behavior remains unchanged.