[Story]: Authenticated OTLP ingestion on otel.ai.camer.digital (AuthConfig #3)

### Story Statement

As a **Foundry agent operator**,
I want to **push OTLP telemetry to an authenticated public endpoint**,
so that **agent activity is recorded in governance without exposing an open ingestion endpoint to the internet.**

### Real Intent

This is the first governance work that touches shared production infrastructure. The feature is ordinary — accept OTLP, authenticate it. The risk is concentrated in one place, and this story exists as much to contain that risk as to deliver the feature.

### Background and Context

There is no ingestion path for telemetry originating outside the cluster.

The shape is already proven here: an Authorino AuthConfig calling a first-party Rust service's resolve endpoint over a shared secret is the ADR-0047 pattern, in production today. What is new is a **third host-indexed AuthConfig** and a **public** host.

### ⚠️ The hazard this story must not trigger

**An AuthConfig `sharedSecretRef` pointing at a Secret that does not exist makes the AuthConfig fail readiness, which 404s the *entire gateway* — every host, every route.**

This is not hypothetical. It is the OPA-removal outage, recorded in `docs/migrations/2026-hetzner-cutover.md`, where pruning an app cascade-deleted the `lightbridge-opa-auth` Secret and took the gateway down.

The mitigation is ordering, not care:

> **The Secret must exist and report `SecretSynced=True` before the AuthConfig references it — in a separate, earlier change.**

### Source of truth (links)

- Epic: #6
- ⚠️ **Blocked by #8** (tenancy spike). Settle the isolation story before writing code.
- Depends on: #11 (`/internal/v1/resolve`), #10 (credentials)
- The outage this must not repeat: `docs/migrations/2026-hetzner-cutover.md`
- Pattern precedent: ai-helm ADR-0047; existing collector notes in `charts/core-gateway/templates/otel.yaml`

### Acceptance Criteria

#### Functional
- [ ] Given a registered integration with a valid credential, when it pushes OTLP to `otel.ai.camer.digital`, then the telemetry is accepted and attributed to its tenant and application.
- [ ] Given telemetry is accepted, when it is processed, then it fans out to metrics, logs and traces.
- [ ] Given a revoked credential, when it pushes, then the request is rejected within the documented cache TTL.

#### Negative / Edge Cases
- [ ] Given **no** credential, when a request is made to the public host, then it is rejected. Verified explicitly — this endpoint is on the internet.
- [ ] Given an invalid or malformed credential, when a request is made, then it is rejected with a single opaque error.
- [ ] Given the governance resolve service is down, when a request arrives, then it is **denied**, not accepted. Fail closed.
- [ ] Given a malformed OTLP payload, when it arrives, then it is rejected without affecting other traffic.

#### Non-Functional
- [ ] ⚠️ **Gateway availability is unaffected throughout rollout.** The AuthConfig change is applied only after its Secret reports `SecretSynced=True`, and a tested revert is ready before it is applied.
- [ ] The added per-request cost of a third host-indexed AuthConfig is **measured** before and after — it is paid by all gateway traffic, including traffic unrelated to Foundry.
- [ ] TLS for `otel.ai.camer.digital` comes from the in-chart namespace ACME **HTTP-01** issuer. ⚠️ **Not** `cert-cloudflare`: `ai.camer.digital` is NS-delegated to Route53, and a DNS-01 challenge there reports `Presented: true` and then stalls forever on "not yet propagated" while cert-manager writes the TXT into the Cloudflare zone. `dig +short NS` first.
- [ ] `X-Scope-OrgID` may be recorded, but is **documented as non-isolating** — it does nothing on our single-tenant LGTM stack.

### Out of Scope

- **Enabling LGTM multi-tenancy.** Decision 3: not now.
- Privacy modes and normalization — related but separate story.
- The integration-setup page.
- Any claim of tenant isolation in customer-facing docs. Whatever #8 decides is what gets said, exactly.

### Dependencies and Blockers

- ⚠️ **Blocked by #8.**
- Depends on #11 being deployed and reachable *before* the AuthConfig references it.
- **Strict ordering, and it is the whole risk mitigation:**
1. Governance resolve service deployed and reachable.
2. Shared-secret Secret created, `SecretSynced=True` **verified**.
3. *Then* the AuthConfig.

Each step is its own change. Do not collapse them.

### Assumptions

1. **Authorino handles a third host-indexed AuthConfig without meaningful overhead.** ⚠️ Measure — do not assume. It is evaluated on every request to the gateway.
2. The ADR-0047 resolve pattern carries over unchanged.
3. OTLP over HTTP is sufficient; gRPC can follow if asked for.
4. A public ingestion endpoint is acceptable given per-integration revocable credentials and rate limiting.

### Implementation Notes

Guidance, not gospel:

- Sequence over care. The ordering above is what prevents the outage; "review carefully" is not a mitigation.
- Have the revert prepared and *tested* before applying the AuthConfig, not written afterwards.
- Rate-limit the public endpoint. Authentication is not a defence against volume.
- Cilium is default-deny; ingress to the collector needs an explicit allow — the observability stack has already lost time to exactly this (`accepted_spans=0` with everything apparently healthy).

### Test Expectations

- Unit: payload validation, credential extraction.
- Integration: authenticated push succeeds; unauthenticated rejected; revoked rejected; resolve-down denies.
- **Manual, in a non-production gateway first:** apply the AuthConfig and confirm other hosts still serve. This is the rehearsal for the failure mode that matters.
- Latency measured on unrelated gateway traffic, before and after.

### Verification evidence

*To be completed on delivery.*

Required:
- `SecretSynced=True` shown for the shared secret **before** the AuthConfig was applied — with timestamps establishing the order.
- Proof that other gateway hosts kept serving during and after the AuthConfig change.
- The negative tests, named individually.
- Before/after latency on unrelated gateway traffic.
- The certificate issued, and confirmation it came from the HTTP-01 issuer.

### Human accountable owner

@stephane-segning

### AI Usage Declaration

Drafting this story, Refining acceptance criteria

### Human verification completed

- [x] I checked the story against the source of truth
- [x] I confirmed the acceptance criteria
- [x] I reviewed the implementation approach
- [x] I documented remaining risks
- [x] I am the accountable owner and accept responsibility for this story.