End-to-end sensor→dashboard latency vs 2s budget: baseline + per-step graphic + instrumentation gap

## Summary

The customer has a **2-second absolute budget** between a sensor taking a measurement and the aggregated value updating on the dashboard. We measured the real end-to-end latency on a live slot (`ws-slot42`) and decomposed it per pipeline step. **RisingWave lands ≈1.33 s of cloud-side + device latency at p50, comfortably under budget; TimescaleDB ≈1.45 s.** The dominant single step is *upstream of any data store* and cannot be reduced by store/RW tuning.

This issue tracks (a) the measured baseline, (b) the per-step graphic, and (c) the instrumentation gap that blocks a *true* per-hop decomposition.

## Measured baseline (ws-slot42, RisingWave `barrier_interval_ms=500`)

| Step | Segment | p50 | How measured |
|---|---|---|---|
| Sensor stamp → IoT Core ingest | device / MQTT hop (**shared, store-independent**) | **~820 ms** | `ingest_ts − message_timestamp` on the source row (n=129) |
| IoT ingest → store visible (RW) | MSK produce + Kafka source read + checkpoint commit | ~480 ms | transition-based: detect when `MAX(ts_ms)` increments |
| IoT ingest → store visible (TSDB) | MSK produce + Redpanda Connect 1 s batch → raw scan | ~600 ms | same technique |
| Store → dashboard | query + SSE push | ~30 ms | dashboard query-latency panel |
| **RisingWave total** | sensor → dashboard | **≈ 1.33 s** (p50) | sum |
| **TimescaleDB total** | sensor → dashboard | **≈ 1.45 s** (p50) | sum |

**Caveats:**
- At **p90** the RW path can exceed the 2 s budget — the ~820 ms device hop has a long tail and poll granularity adds up to a barrier interval.
- The ~820 ms device→ingest figure may include a small amount of device/IoT clock skew, though EC2 hosts are NTP-synced so it is likely mostly real MQTT→IoT-rule transit.
- Lowering `barrier_interval_ms` 1000→500 roughly **halved** the RW cloud-side portion (freshness ~3 s → ~1.3 s) and was validated to cause **zero ingestion backlog** on r6i.xlarge.

## Per-step graphic

A waterfall graphic of the absolute increment per step (both stores, with the 2 s budget line) is at `tmp/progress/2026-08-14-latency-waterfall.svg` / `.png` in the repo. Rendered summary:

```
0 0.5s 1.0s 1.5s 2.0s budget
RisingWave [==== 820 device ====][== 480 RW ==]|~30| ≈1.33s ✓
TimescaleDB [==== 820 device ====][== ~600 TSDB ==]|~30| ≈1.45s
┊ 2.0s
```

**Key takeaway for the customer story:** the ~820 ms device→ingest hop is the largest single step, is identical for every store, and is ~40 % of the 2 s budget on its own. Store choice (RW vs TSDB) only moves the smaller cloud-side segment.

## Instrumentation gap (the actual work item)

The per-hop numbers above are inferred from two timestamps we happen to have (`message_timestamp` device-stamped, `ts_ms`/`ingest_ts` IoT-Core-stamped) plus transition detection. We **cannot** currently isolate MSK-produce vs source-read vs commit, because **no processing-time stamps are injected at the MSK-consume and sink-write boundaries**. To get a defensible per-hop breakdown we need to:

- [ ] Stamp a processing-time column when a record is produced to MSK (IoT Rule or a lightweight transform).
- [ ] Stamp processing-time at sink-write (RW MV compute time; TSDB insert time / Redpanda Connect).
- [ ] Surface these on the dashboard freshness panel so the decomposition is live, not a one-off manual measurement.
- [ ] Re-run the waterfall from real per-boundary stamps and replace the inferred segments above.

## Acceptance

- Waterfall graphic committed somewhere durable (workshop docs asset or reference/), not just `tmp/`.
- Documented statement of the p50/p90 sensor→dashboard latency vs the 2 s budget for both stores.
- Instrumentation checklist above either implemented or explicitly deferred with rationale.

Start with the existing tmp/progress/2026-08-14-latency-waterfall.svg and .png and review the instrumentation checklist across the IoT Rule, MSK boundaries, RisingWave MV, TimescaleDB/Redpanda Connect sink, and dashboard freshness panel. Done means a durable waterfall asset, documented p50/p90 latency for both stores, and each instrumentation item implemented or explicitly deferred with rationale.

Written by the indexing model from the issue text.