aws-samples / aws-samples/sample-edge-to-cloud-digital-ops-workshop
Epic: hit the 2s sensor→dashboard latency budget + make per-step latency live
- Dominant language
- TypeScript
- Stars
- 3
- Forks
- 1
- Avg merge
- 7h 19m
- Merged PRs (30d)
- 12
Description
## Epic: Hit the 2-second sensor→dashboard latency budget, and make the per-step latency visible live
The customer has a **2 s absolute budget** from sensor measurement to an aggregated value updating on the dashboard. We measured the live pipeline on `ws-slot42` and decomposed it per step (baseline in #242, RW tuning findings in #243). Two actionable conclusions came out of it:
1. **The single biggest latency source is the device publish mechanism**, not network transit. The device shells out to `aws iot-data publish` (the Python AWS CLI, an HTTPS REST call) as a **brand-new process for every message** — paying Python cold-start + credential resolution + a fresh TLS handshake each time. That's the ~820 ms `message_timestamp → ingest_ts` gap (n=129, p50), ~40 % of the whole budget, and it is entirely device-side overhead. See [job-scripts/telemetry-v2.sh](https://github.com/aws-samples/sample-edge-to-cloud-digital-ops-workshop/blob/main/job-scripts/telemetry-v2.sh).
2. **Almost the entire per-step waterfall can be shown live on the dashboard** with existing data. The ingest→visible bar (freshness) and store→dashboard bar (query latency) are already live; the device-hop bar just needs a small MV/query change because both `message_timestamp` and `ingest_ts` already flow into RisingWave.
### Scope of this epic (one PR)
| # | Work | Expected impact |
|---|---|---|
| A (new) | **Device: persistent MQTT publisher** replacing per-message `aws iot-data publish` | ~820 ms → ~20–80 ms; the ~750 ms win |
| B (new) | **Live device-hop latency panel** (`ingest_ts − message_timestamp`) on the cloud dashboard | makes the dominant segment visible; no new instrumentation |
| #243 | **Make `barrierIntervalMs: 500` permanent** in `helm/cloud-analytics/values.yaml` | halves RW cloud-side latency; validated safe |
### Explicitly OUT of scope (deferred, stays in #242)
The finer decomposition *within* the ingest→visible segment (MSK-produce vs Kafka-source-read vs checkpoint-commit) requires injecting processing-time stamps at the MSK-consume and sink-write boundaries. That is a larger, live-AWS-heavy change tracked as the instrumentation-gap checklist in #242 — do **not** attempt it in this epic.
### Related / context
- #242 — end-to-end latency baseline, per-step graphic, instrumentation gap
- #243 — Kafka→RisingWave connector tuning (barrier interval is the lever; fetch-backoff refuted)
### Definition of done for the epic
- One PR closing sub-issues A, B, and the barrier item of #243.
- `mkdocs build --strict` passes; `cloud-dashboard` builds; `helm template helm/cloud-analytics` renders.
- Live-slot validation (does the MQTT publisher actually cut the hop; does the panel show real numbers) is done by the **maintainer's local session**, not the agent — the agent's job is a building, review-clean implementation.
### Tracking
- [x] #245 — live device→ingest panel — **validated on ws-slot42** (reads real number, null-degrades)
- [x] #243 — `barrierIntervalMs: 500` permanent — **deployed** (RW ingest-lag ~568 ms, consistent with 500 ms barrier)
- [ ] #244 — device persistent MQTT publisher — implemented but **does not activate on deployed devices**; blocked by #248
- [ ] #248 — fix: publisher never installs (S3 upload race + UserData abort) — **dispatched to remote agent**
Contributor guide
Research direction
Start by reading job-scripts/telemetry-v2.sh and the implementation and deployment context in #244 and #248; review helm/cloud-analytics/values.yaml and the existing cloud-dashboard entry points for the live latency panel. Done means the persistent publisher, device-hop panel, and permanent barrier setting are implemented in one review-clean PR, with mkdocs build --strict, the dashboard build, and helm template helm/cloud-analytics passing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, helm, kafka, typescript
- Domain
- cloud, devops, frontend, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100