aws-samples / aws-samples/sample-edge-to-cloud-digital-ops-workshop

Epic: hit the 2s sensor→dashboard latency budget + make per-step latency live

Open
#246 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
3
Forks
1
Avg merge
7h 19m
Merged PRs (30d)
12

Description

## Epic: Hit the 2-second sensor→dashboard latency budget, and make the per-step latency visible live

The customer has a **2 s absolute budget** from sensor measurement to an aggregated value updating on the dashboard. We measured the live pipeline on `ws-slot42` and decomposed it per step (baseline in #242, RW tuning findings in #243). Two actionable conclusions came out of it:

1. **The single biggest latency source is the device publish mechanism**, not network transit. The device shells out to `aws iot-data publish` (the Python AWS CLI, an HTTPS REST call) as a **brand-new process for every message** — paying Python cold-start + credential resolution + a fresh TLS handshake each time. That's the ~820 ms `message_timestamp → ingest_ts` gap (n=129, p50), ~40 % of the whole budget, and it is entirely device-side overhead. See [job-scripts/telemetry-v2.sh](https://github.com/aws-samples/sample-edge-to-cloud-digital-ops-workshop/blob/main/job-scripts/telemetry-v2.sh).

2. **Almost the entire per-step waterfall can be shown live on the dashboard** with existing data. The ingest→visible bar (freshness) and store→dashboard bar (query latency) are already live; the device-hop bar just needs a small MV/query change because both `message_timestamp` and `ingest_ts` already flow into RisingWave.

### Scope of this epic (one PR)

| # | Work | Expected impact |
|---|---|---|
| A (new) | **Device: persistent MQTT publisher** replacing per-message `aws iot-data publish` | ~820 ms → ~20–80 ms; the ~750 ms win |
| B (new) | **Live device-hop latency panel** (`ingest_ts − message_timestamp`) on the cloud dashboard | makes the dominant segment visible; no new instrumentation |
| #243 | **Make `barrierIntervalMs: 500` permanent** in `helm/cloud-analytics/values.yaml` | halves RW cloud-side latency; validated safe |

### Explicitly OUT of scope (deferred, stays in #242)

The finer decomposition *within* the ingest→visible segment (MSK-produce vs Kafka-source-read vs checkpoint-commit) requires injecting processing-time stamps at the MSK-consume and sink-write boundaries. That is a larger, live-AWS-heavy change tracked as the instrumentation-gap checklist in #242 — do **not** attempt it in this epic.

### Related / context
- #242 — end-to-end latency baseline, per-step graphic, instrumentation gap
- #243 — Kafka→RisingWave connector tuning (barrier interval is the lever; fetch-backoff refuted)

### Definition of done for the epic
- One PR closing sub-issues A, B, and the barrier item of #243.
- `mkdocs build --strict` passes; `cloud-dashboard` builds; `helm template helm/cloud-analytics` renders.
- Live-slot validation (does the MQTT publisher actually cut the hop; does the panel show real numbers) is done by the **maintainer's local session**, not the agent — the agent's job is a building, review-clean implementation.

### Tracking
- [x] #245 — live device→ingest panel — **validated on ws-slot42** (reads real number, null-degrades)
- [x] #243 — `barrierIntervalMs: 500` permanent — **deployed** (RW ingest-lag ~568 ms, consistent with 500 ms barrier)
- [ ] #244 — device persistent MQTT publisher — implemented but **does not activate on deployed devices**; blocked by #248
- [ ] #248 — fix: publisher never installs (S3 upload race + UserData abort) — **dispatched to remote agent**

Contributor guide

Open the contributing guide

Research direction

Start by reading job-scripts/telemetry-v2.sh and the implementation and deployment context in #244 and #248; review helm/cloud-analytics/values.yaml and the existing cloud-dashboard entry points for the live latency panel. Done means the persistent publisher, device-hop panel, and permanent barrier setting are implemented in one review-clean PR, with mkdocs build --strict, the dashboard build, and helm template helm/cloud-analytics passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, helm, kafka, typescript
Domain
cloud, devops, frontend, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.