[Story]: Logs investigation in apps/console — why did my telemetry stop arriving?

## 1. Summary

This story adds:

- A **logs investigation surface in `apps/console`** — a tenant-scoped view answering *"what did my integration actually send, and why was it refused?"*
- A **narrow, purpose-built query endpoint** on the authz side that it reads through the console's existing Next proxy.

It solves:

- There is no product surface for log investigation. Logs live in Loki and are reachable only through Grafana — operator territory, not tenant-facing. When an integration's telemetry stops arriving, the tenant has nowhere in the product to look.

---

## 2. Intent

The intent of this PR is:

> When telemetry stops arriving, the person who owns that integration should be able to see *why* in the console — not file a ticket and wait for someone with Grafana access to read it for them.

This is the one leg of the observability story with no issue behind it. Dashboards are covered three ways — [lightbridge-authz#508](https://github.com/ADORSYS-GIS/lightbridge-authz/issues/508) (console Overview usage graphs), [converse-frontends#327](https://github.com/ADORSYS-GIS/converse-frontends/issues/327) (the leadership dashboards app), [ai-helm#1080](https://github.com/ADORSYS-GIS/ai-helm/issues/1080) (operator Grafana). **Investigation is covered zero ways.**

The concrete case that motivated it, from 2026-09-02: the `ai-cli` collector logged `Authentication failed: missing or empty header` every 5–20 seconds for days. Answering *"who?"* took a live `kubectl` investigation across three namespaces, and the answer is still inferred rather than observed. A tenant cannot do that, and should not have to.

---

## 3. Scope

### In Scope

- A logs view in `apps/console`, **scoped to the caller's tenant** — recent ingest events for their integrations: accepted, refused, and why.
- A **narrow query endpoint** on the authz side, purpose-built for this view.
- Read path through the console's existing Next proxy (`apps/console/src/app/api/usage/[...path]/route.ts` is the established pattern, from #508's chain).
- Honest unavailable states, per the precedent this repo already set for usage panels.

### Out of Scope

- **Proxying Loki from the console.** See Risk Assessment — this is the design decision the story turns on, and it is deliberately excluded.
- Free-text log search / LogQL. This is bounded investigation, not a log explorer.
- Operator-wide or cross-tenant views — [lightbridge-authz#602](https://github.com/ADORSYS-GIS/lightbridge-authz/issues/602) covers operator-privileged enumeration.
- The dashboards themselves (#508, #327, ai-helm#1080).

---

## 4. Verification

I verified this change by:

- [ ] Running automated tests
- [ ] Running manual tests
- [x] Checking logs
- [ ] Checking metrics
- [x] Testing error cases
- [x] Testing permissions/security behavior
- [ ] Testing rollback or failure behavior, if relevant

**Acceptance criteria** — this is a story, so these are what "verified" will mean:

1. A tenant sees their own integrations' recent ingest events, with the refusal reason where one exists.
2. **Tenant isolation is enforced server-side**, not by a client-side filter. A test asserts a caller cannot read another tenant's events by manipulating the request.
3. The view degrades honestly when the backing store is unavailable — an unavailable state, never a fake empty list. (The console already has this precedent for usage panels; reuse it.)
4. The endpoint is **bounded**: a time window and a row cap, so no query can full-scan. ⚠️ [lightbridge-authz#606](https://github.com/ADORSYS-GIS/lightbridge-authz/issues/606) records a real 13–25s full-scan on `usage_events` from an unbounded pre-query — do not add a second.
5. Every new or changed file under `src/` stays at or below 200 LoC (this repo's standing rule, restated in #508).
6. **Falsification:** remove the server-side tenant scope and confirm the isolation test fails.

---

## 5. Screenshots / Evidence

The failure this makes visible, from the `ai-cli` collector on 2026-09-02:

```
warn oidcauthextension Authentication failed: missing or empty header
{"client_ip": "10.42.1.63", "error": "authentication didn't succeed"}
```

Answering *"which client?"* required `kubectl get pods -A -o wide` across namespaces — and `client_ip` turned out to be a Traefik pod, not the caller ([ai-helm#1081](https://github.com/ADORSYS-GIS/ai-helm/issues/1081) fixes that separately). This story is what makes the tenant-visible half of that question answerable in-product.

---

## 6. Risk Assessment

Risk level:

* [ ] Low
* [x] Medium
* [ ] High

Potential risks:

* ⚠️ **The data path is an open decision, and it shapes the whole story.** The console has no Loki access today. Per [#581](https://github.com/ADORSYS-GIS/lightbridge-authz/issues/581)'s design the *governed* store holds normalized usage rows, not raw log lines — so a console logs view either proxies Loki (new dependency, new auth surface, tenant-scoping a store that was never designed for it) or reads a purpose-built endpoint over the governed store.
* Refusal reasons may not be captured anywhere queryable today — refused pushes fail at the collector's OIDC extension *before* the pipeline, so they never reach the governed store at all (`otelcol_receiver_refused_* = 0` while auth failures logged continuously). **The most interesting events may be exactly the ones no store currently holds.**
* Log content is prompt-adjacent. Anything surfaced must respect the same posture as the rest of the platform: no prompts, no completions, no tokens.

Mitigation:

* This story proposes the **narrow endpoint**, not the Loki proxy — smaller blast radius, tenant scoping designed in rather than retrofitted, and no new infrastructure dependency for the console.
* The refusal-visibility gap is called out above rather than assumed away. If refused events must be visible, capturing them is a **prerequisite** and probably its own ticket — resolve that before estimating this one.

---

## 7. AI Usage Declaration

AI was used for:

* [x] Understanding existing code
* [ ] Generating code
* [ ] Refactoring
* [ ] Generating tests
* [x] Drafting documentation
* [ ] Reviewing the diff
* [ ] Not used

Filed by Claude Opus 5 at the maintainer's direction, after searching `converse-frontends`, `lightbridge-authz` and `lightbridge-governance` and confirming no logs-investigation story exists — dashboards are covered three times over, investigation not at all. The collector evidence quoted above is from a live `kubectl` investigation on 2026-09-02, not inferred.

Human verification:

* [ ] I understand every meaningful change in this PR
* [ ] I checked generated code manually
* [ ] I checked generated tests manually
* [x] I removed unsupported AI assumptions
* [ ] I accept responsibility for this PR — *for the human author*

---

## 8. Reviewer Focus

Please focus your review on:

* [x] Architecture
* [x] Security
* [x] Product intent
* [x] Edge cases
* [ ] Performance
* [ ] Maintainability
* [ ] Correctness
* [ ] Tests

Specifically:

1. **Narrow endpoint vs Loki proxy.** The story assumes the former. If you want real log lines rather than ingest events, that is a different and larger story — say so before it is estimated.
2. **Are refused events in scope?** They are the most useful thing to investigate and the least likely to be stored. That answer changes the size materially.
3. **Console vs the #327 dashboards app.** Console is chosen because its audience already owns keys, budgets and projects; #327's charter is leadership KPIs, which log spelunking does not fit.