Real-time runtime security alerts and full log aggregation in CloudWatch

The workshop proves runtime enforcement — Vault denies the forged token, the wrong agent, the wrong RAR path — but nobody is paged while it happens. And most components' logs never leave the cluster: fluent-bit tails every pod but ships only three feeds (`vault-*`, `iviaop-*`, `uc3-agent-*`); banking-ui/agent/mcp-server, uc1-agent, openldap and four of five IVIA pods are read and discarded, with the IVIA four also losing their app logs to an emptyDir on restart.

Scope:

- **Aggregate**: catch-all fluent-bit output so every workload pod's stdout lands in a `/workshop/apps` CloudWatch log group (stream per pod), alongside the existing `/workshop/vault-audit`, `/workshop/ivia-decision`, `/workshop/agent-trace`, and RDS pgaudit groups — everything viewable in AWS-native CloudWatch Logs seconds after emission.
- **Alert**: two consumers of one signal definition — metric filters + CloudWatch alarms + SNS (near-real-time, ~60-120s: the stateful red/green tile surface) AND a second subscription filter → Lambda → SNS (seconds-level pager, measured ~2-5s). Signals are the exact abuse signals the Bypass Test page produces — a denied credential read on `database/creds/uc3-refund-writer` in the Vault audit stream (forged signature, wrong agent, wrong RAR all land here), and IVIA's `unauthorized_client` token-exchange refusal (the structured `level=AUDIT` record — NOT the `FBTAQ50xx` codes, which appear in stdout only as catalog-load lines at startup and would alarm on every pod restart). An attendee runs the bypass page and watches their own attack raise the alarm.
- **View**: a saved Logs Insights query reproducing the three-plane correlation live in CloudWatch (Athena stays as the forensic plane), plus a `-runtime-security` CloudWatch dashboard — alarm tiles, denial counts, latest-denial tables, per-plane log volume on one auto-refreshing page.

## How to test

All commands below were run live on 2026-09-03 (branch `feat/44-cloudwatch-runtime-alerts`, cluster `ars-workshop`, us-east-1) with the stated results.

1. Run the Bypass Test page's own command — it produces every attack the alarms watch (Check 14 forged HS256, Check 18 wrong-agent deny, Check 20 `unauthorized_client`):

```bash
cd infrastructure/scripts && UC3_PERSONA=jaime ./verify-uc3.sh --bypass
```

Expected (seen live): `✓ 8 check(s) passed`, one deliberate Check 19 SKIP. Note `UC3_PERSONA=jaime` — persona `oscar` currently has a REAL IBM Verify device enrolled (biometrics key, no tool marker), and the mint guard correctly refuses to race it. That refusal is the guard working, not a failure.

2. Watch both alarms fire, then recover:

```bash
aws cloudwatch describe-alarms --region us-east-1 --alarm-name-prefix ars-workshop \
--query 'MetricAlarms[].{Name:AlarmName,State:StateValue}' --output table
```

Expected (seen live): `ars-workshop-vault-refund-cred-denied` → ALARM within ~60s of the denials, back to OK the next quiet minute; `ars-workshop-ivia-exchange-refused` → ALARM off Check 20. Two full OK→ALARM→OK cycles are in each alarm's history (21:15/21:21 and 21:16/21:22 UTC). Alarm history shows `Successfully executed action arn:aws:sns:…:ars-workshop-security-alerts` on every transition.

3. Catch-all aggregation — app pods land in `/workshop/apps`, dedicated feeds are NOT duplicated:

```bash
aws logs describe-log-streams --region us-east-1 --log-group-name /workshop/apps \
--order-by LastEventTime --descending --max-items 10 \
--query 'logStreams[].logStreamName' --output table
```

Expected (seen live): streams for banking-app, verify-access (iviadsc, openldap, iviaconfig…), kube-system pods — and ZERO streams prefixed `apps-kube.var.log.containers.vault-`, `…iviaop-`, `…uc3-agent-`, or `…aws-for-fluent-bit-` (self-logs excluded; verified no new self-log events after the corrected DaemonSet rolled).

4. Three-plane live trace — Logs Insights → Saved queries → `workshop/trace-request-id-across-planes`, substitute a real refund `request_id` for `REQUEST_ID`. Seen live with `request_id=2f42eedf-…`: 18 rows spanning `/workshop/agent-trace`, `/workshop/vault-audit`, `/workshop/ivia-decision` in emission order.

5. Negative test / pipeline regression — the change must not disturb the refund path:

```bash
cd infrastructure/scripts && ./verify-uc3.sh --no-phone
```

Expected (seen live): `✓ 8 check(s) passed` — real refund $88.30 for jaime, `request_id=d12ed1bc-d4c1-4eeb-a89e-466b11192e1e`, Athena `audit_correlation` 12 columns / 0 blanks.

6. To get paged for real, subscribe to the topic (nothing is subscribed yet by design):

```bash
aws sns subscribe --region us-east-1 \
--topic-arn $(terraform -chdir=infrastructure output -raw security_alerts_topic_arn) \
--protocol email --notification-endpoint you@example.com
```

7. The dashboard — one auto-refreshing pane instead of digging through logs:

```bash
terraform -chdir=infrastructure output -raw security_dashboard_url
```

Open the URL (CloudWatch → Dashboards → `ars-workshop-runtime-security`). Expected (seen live in the console mid-attack): both alarm tiles red during a bypass run, denial spikes on the per-minute graph, the latest Vault credential denials (path + JWT display name + plaintext deny reason) and IVIA refusals (cause, client, origin) as tables, and log volume flowing on all five planes. The Vault table filters to `database/creds/*` — without that filter, routine `auth/token/lookup-self` errors pollute it (found and fixed live).

8. Seconds-level push — the Lambda pager, measured live (denial record timestamp → SNS MessageId in the function log):

```bash
aws logs tail /aws/lambda/ars-workshop-security-push --region us-east-1 --since 15m --format short
```

Expected (seen live): one `{"published": "", "events": 1, "logGroup": "/workshop/..."}` line per denial. Measured deltas: forged-HS256 ~2s, wrong-agent ~3s, IVIA `unauthorized_client` ~5s after the denial record's own timestamp. Subscribe an email to the topic (step 6) to receive the plain-English message itself.

9. Workshop content — the new optional page 'Get Paged When the Attack Happens' (`workshop/content/70-use-case-3/72-runtime-alerts/`, weight 72.5, right before The Bypass Test) walks an attendee through subscribing their inbox and opening the dashboard BEFORE their one bypass run, so that run is the live detection demo. Every command on the page was run verbatim (subscribe → `{"SubscriptionArn": "pending confirmation"}`, list → `PendingConfirmation` then a full ARN after confirming, dashboard URL from `terraform output`). cspell + prettier clean; The Bypass Test page gained one pointer line.