[Story]: Authenticated OTLP ingestion on otel.ai.camer.digital (AuthConfig #3)

### Story Statement

As a **Foundry agent operator**,
I want to **push OTLP telemetry to an authenticated public endpoint**,
so that **agent activity is recorded in governance without exposing an open ingestion endpoint to the internet.**

### Real Intent

This is the first governance work that touches shared production infrastructure. The feature is ordinary — accept OTLP, authenticate it. The risk is concentrated in one place, and this story exists as much to contain that risk as to deliver the feature.

### Background and Context

There is no ingestion path for telemetry originating outside the cluster.

The shape is already proven here: an Authorino AuthConfig calling a first-party Rust service's resolve endpoint over a shared secret is the ADR-0047 pattern, in production today. What is new is a **third host-indexed AuthConfig** and a **public** host.

### ⚠️ The hazard this story must not trigger

**An AuthConfig `sharedSecretRef` pointing at a Secret that does not exist makes the AuthConfig fail readiness, which 404s the *entire gateway* — every host, every route.**

This is not hypothetical. It is the OPA-removal outage, recorded in `docs/migrations/2026-hetzner-cutover.md`, where pruning an app cascade-deleted the `lightbridge-opa-auth` Secret and took the gateway down.

The mitigation is ordering, not care:

> **The Secret must exist and report `SecretSynced=True` before the AuthConfig references it — in a separate, earlier change.**

### Source of truth (links)

- Epic: #6
- ⚠️ **Blocked by #8** (tenancy spike). Settle the isolation story before writing code.
- Depends on: #11 (`/internal/v1/resolve`), #10 (credentials)
- The outage this must not repeat: `docs/migrations/2026-hetzner-cutover.md`
- Pattern precedent: ai-helm ADR-0047; existing collector notes in `charts/core-gateway/templates/otel.yaml`

### Acceptance Criteria

#### Functional
- [ ] Given a registered integration with a valid credential, when it pushes OTLP to `otel.ai.camer.digital`, then the telemetry is accepted and attributed to its tenant and application.
- [ ] Given telemetry is accepted, when it is processed, then it fans out to metrics, logs and traces.
- [ ] Given a revoked credential, when it pushes, then the request is rejected within the documented cache TTL.

#### Negative / Edge Cases
- [ ] Given **no** credential, when a request is made to the public host, then it is rejected. Verified explicitly — this endpoint is on the internet.
- [ ] Given an invalid or malformed credential, when a request is made, then it is rejected with a single opaque error.
- [ ] Given the governance resolve service is down, when a request arrives, then it is **denied**, not accepted. Fail closed.
- [ ] Given a malformed OTLP payload, when it arrives, then it is rejected without affecting other traffic.

#### Non-Functional
- [ ] ⚠️ **Gateway availability is unaffected throughout rollout.** The AuthConfig change is applied only after its Secret reports `SecretSynced=True`, and a tested revert is ready before it is applied.
- [ ] The added per-request cost of a third host-indexed AuthConfig is **measured** before and after — it is paid by all gateway traffic, including traffic unrelated to Foundry.
- [ ] TLS for `otel.ai.camer.digital` comes from the in-chart namespace ACME **HTTP-01** issuer. ⚠️ **Not** `cert-cloudflare`: `ai.camer.digital` is NS-delegated to Route53, and a DNS-01 challenge there reports `Presented: true` and then stalls forever on "not yet propagated" while cert-manager writes the TXT into the Cloudflare zone. `dig +short NS` first.
- [ ] `X-Scope-OrgID` may be recorded, but is **documented as non-isolating** — it does nothing on our single-tenant LGTM stack.

### Out of Scope

- **Enabling LGTM multi-tenancy.** Decision 3: not now.
- Privacy modes and normalization — related but separate story.
- The integration-setup page.
- Any claim of tenant isolation in customer-facing docs. Whatever #8 decides is what gets said, exactly.

### Dependencies and Blockers

- ⚠️ **Blocked by #8.**
- Depends on #11 being deployed and reachable *before* the AuthConfig references it.
- **Strict ordering, and it is the whole risk mitigation:**
1. Governance resolve service deployed and reachable.
2. Shared-secret Secret created, `SecretSynced=True` **verified**.
3. *Then* the AuthConfig.

Each step is its own change. Do not collapse them.

### Assumptions

1. **Authorino handles a third host-indexed AuthConfig without meaningful overhead.** ⚠️ Measure — do not assume. It is evaluated on every request to the gateway.
2. The ADR-0047 resolve pattern carries over unchanged.
3. OTLP over HTTP is sufficient; gRPC can follow if asked for.
4. A public ingestion endpoint is acceptable given per-integration revocable credentials and rate limiting.

### Implementation Notes

Guidance, not gospel:

- Sequence over care. The ordering above is what prevents the outage; "review carefully" is not a mitigation.
- Have the revert prepared and *tested* before applying the AuthConfig, not written afterwards.
- Rate-limit the public endpoint. Authentication is not a defence against volume.
- Cilium is default-deny; ingress to the collector needs an explicit allow — the observability stack has already lost time to exactly this (`accepted_spans=0` with everything apparently healthy).

### Test Expectations

- Unit: payload validation, credential extraction.
- Integration: authenticated push succeeds; unauthenticated rejected; revoked rejected; resolve-down denies.
- **Manual, in a non-production gateway first:** apply the AuthConfig and confirm other hosts still serve. This is the rehearsal for the failure mode that matters.
- Latency measured on unrelated gateway traffic, before and after.

### Verification evidence

*To be completed on delivery.*

Required:
- `SecretSynced=True` shown for the shared secret **before** the AuthConfig was applied — with timestamps establishing the order.
- Proof that other gateway hosts kept serving during and after the AuthConfig change.
- The negative tests, named individually.
- Before/after latency on unrelated gateway traffic.
- The certificate issued, and confirmation it came from the HTTP-01 issuer.

### Human accountable owner

@stephane-segning

### AI Usage Declaration

Drafting this story, Refining acceptance criteria

### Human verification completed

- [x] I checked the story against the source of truth
- [x] I confirmed the acceptance criteria
- [x] I reviewed the implementation approach
- [x] I documented remaining risks
- [x] I am the accountable owner and accept responsibility for this story.

Start with blocker #8, then read the ADR-0047 precedent, charts/core-gateway/templates/otel.yaml, and docs/migrations/2026-hetzner-cutover.md; verify the /internal/v1/resolve dependency and SecretSynced ordering. Run the stated unit, integration, non-production gateway, TLS, and latency checks. Done means authenticated OTLP works, unauthenticated and failure cases deny safely, other hosts remain available, and the required evidence is recorded.

Vom Indexierungsmodell aus dem Issue-Text verfasst.