[Epic]: Microsoft Foundry connector — authenticated OTLP telemetry ingestion

### Executive Summary

We want **authenticated OTLP ingestion of Microsoft Foundry agent telemetry**, so that AI usage running on customer-side Foundry agents becomes visible in the same governance surfaces as Copilot usage and gateway usage.

Unlike the Copilot connector, this one is **push**: a public `otel.` host, an Authorino AuthConfig, and a collector that fans telemetry out to Mimir, Loki and Tempo. That makes it the first governance epic that touches shared production infrastructure, and its risk sits almost entirely in one YAML field.

### Strategic Intent

Accept AI telemetry pushed from outside the cluster, authenticate it per-integration, and land it in the observability stack — without letting a misconfigured AuthConfig take the gateway down.

### Problem Statement

- **Users / governance team:** Foundry agent activity is entirely unobserved. There is no ingestion path for telemetry that originates outside our cluster.
- **Developers / operations:** the spec's isolation model assumes a multi-tenant LGTM stack. **Ours is single-tenant** — verified in `ai-helm-values`: `mimir.yaml` `multitenancy_enabled: false`, `loki.yaml` `auth_enabled: false`, `tempo.yaml` `replicas: 1` (single-binary). `mimir.yaml` even carries the comment "a multi-tenant fleet we don't run." So `X-Scope-OrgID` currently does **nothing**, and the spec's "tenant A cannot query tenant B" matrix is not a config tweak — it is a migration of the stack that monitors our own production.
- **Security / compliance:** a public ingestion endpoint is new attack surface and needs authentication that can be revoked per-integration.

### Desired Outcome

- Foundry agents can push OTLP to `otel.ai.camer.digital`, authenticated per-integration against the registry.
- Telemetry fans out to metrics, logs and traces, normalized to the governance schema and to integer micro-USD.
- Privacy modes are enforced at ingestion, so what we do not want to store is never stored.
- **The gateway does not go down at any point during rollout.**

### Scope (In / Out)

#### In Scope
- `otel.ai.camer.digital` on core-gateway (public LB, ACME HTTP-01) — the in-chart namespace ACME Issuer + `gatewayHTTPRoute` solver, since `cert-cloudflare` cannot issue for NS-delegated `ai.camer.digital`.
- **Authorino AuthConfig #3**, host-indexed alongside `main` and `internal`, calling the governance service's resolve endpoint over a shared secret.
- Collector + three-signal fan-out (Mimir / Loki / Tempo).
- Privacy modes and payload normalization.
- Per-integration credential resolution against the registry (#4).

#### Out of Scope
- **Turning on LGTM multi-tenancy.** Decision 3: not now. Single tenant for us; a customer decides for their own install. `X-Scope-OrgID` is accepted and recorded, but is not an isolation boundary today, and **must not be documented as one**.
- **A Next.js console.** Governance views are Grafana; only the integration-setup page needs bespoke UI, and that is one server-rendered page.
- The tenancy end state (option B). Recommendation is option C now, B later — settled by the Wave 0 tenancy spike.

### The one thing that can break production

⚠️ **An AuthConfig `sharedSecretRef` pointing at a Secret that does not exist makes the AuthConfig fail readiness, which 404s the *entire gateway* — every host, every route.**

This is not hypothetical: it is the OPA-removal outage, already documented in `docs/migrations/2026-hetzner-cutover.md`. The mitigation is not "be careful" but a hard ordering rule:

> **The Secret must exist and report `SecretSynced=True` *before* the AuthConfig references it.** Secret-first, always, in a separate change.

Also not free, and worth stating plainly: Authorino evaluates the AuthConfig on **every** request to the gateway, so a third host-indexed config is a per-request cost on traffic that has nothing to do with Foundry.

### Good news

**Increment 2 of the source spec is already built** — not "something similar", the exact flow: an Authorino AuthConfig calling a first-party Rust service's `/v1/resolve` over a shared secret. That is the ADR-0047 pattern, running in production today. This epic reuses it rather than inventing it.

### Source of truth (links)

- Roadmap: ADORSYS-GIS/ai-helm#872, §4 Wave 3 and §3 (risk ranking — this is a 🟡 row *because of* the AuthConfig edit)
- Full plan: `plans/microsoft-foundry-governance.md` in that PR, §0 (the headline finding) and §3 (the tenancy decision)
- Source spec, in-repo: `docs/rfc/sources/`
- Depends on: #4 (registry) — per-integration credentials resolve against it
- Gated by: the LGTM tenancy spike (Wave 0) — **settle before writing code**
- Prior outage this epic must not repeat: `docs/migrations/2026-hetzner-cutover.md` (OPA removal / missing `lightbridge-opa-auth` Secret → gateway-wide 404)
- Existing OTel collector notes: `charts/core-gateway/templates/otel.yaml`

### Stakeholders

- Product Owner / Technical Lead / Security: @stephane-segning

### Key Assumptions

1. **LGTM stays single-tenant for the MVP** (decision 3). If that changes, this epic's ingestion path is unaffected but its isolation claims change entirely.
2. **The ADR-0047 resolve pattern carries over unchanged.** Assumed proven — it is in production.
3. **A third host-indexed AuthConfig is acceptable per-request overhead.** Should be measured, not assumed, before rollout.
4. `otel.ai.camer.digital` can get an ACME HTTP-01 certificate. ⚠️ It **cannot** use `cert-cloudflare` — `ai.camer.digital` is NS-delegated to Route53, and a DNS-01 challenge there stalls forever on "not yet propagated" while cert-manager writes TXT records into the wrong zone.

### Constraints

- Secret-first ordering on anything the AuthConfig references. Non-negotiable.
- No LGTM multi-tenancy.
- `X-Scope-OrgID` may be recorded but must not be described as isolation.
- Integer micro-USD for money.
- Values-repo-first for anything reaching `ai-helm-values`.

### Risks

| Risk | Probability | Impact | Mitigation |
|---|---|---|---|
| **AuthConfig references a missing Secret → gateway-wide 404** | Medium | 🔴 **Critical** | Secret-first in a separate change; verify `SecretSynced=True`; have the revert ready before applying |
| Someone documents `X-Scope-OrgID` as an isolation boundary | **High** | High | Say plainly in the ADR and the integration docs that it is inert today |
| Public ingestion endpoint abused | Medium | Medium | Per-integration credentials, revocable via the registry; rate limiting |
| Third AuthConfig adds per-request latency to all gateway traffic | Medium | Medium | Measure before and after; this is a cost paid by unrelated traffic |
| DNS-01 attempted for the cert and stalls silently | Medium | Low | Use the in-chart ACME HTTP-01 issuer; `dig +short NS` first |

### Success metrics

| Metric | Current | Target | Source |
|---|---|---|---|
| Foundry telemetry ingested | none | all registered integrations | governance DB / Mimir |
| Gateway availability during rollout | 100% | **100%** | uptime; this is the metric that matters |
| Time to revoke a compromised integration credential | n/a | ≤ documented cache TTL | integration test |
| Telemetry rejected without valid credentials | n/a | 100% | negative test |

### Child User Stories

- Spike: LGTM tenancy decision — A, B or C *(blocks this epic)*
- `otel.` host + ACME HTTP-01 certificate *(ai-helm)*
- ⚠️ Shared secret provisioning, **secret-first, separate change** *(ai-helm-values)*
- Authorino AuthConfig #3 + resolve integration *(ai-helm)*
- Collector + three-signal fan-out
- Privacy modes + normalization
- Integration-setup page

### Human accountable owner

@stephane-segning

### AI Usage Declaration

Drafting, Research, Ticket decomposition, Technical proposal

### Human verification completed

- [x] Intent checked against source of truth
- [x] Risks reviewed
- [x] I am the accountable owner and accept responsibility for this epic.