aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

Add CloudWatch alarm for ApprovalMetricsPublisher EMF rate ceiling

Aperta Adatta ai principianti
#114 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
observability
Lingua principale
TypeScript
Stelle
143
Fork
46
Merge medio
3g 10h
PR unite (30g)
24

Descrizione

> **Follow-up from [PR #88](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/pull/88)** — observability blind spot in the metrics publisher path.

## Functional description

The `ApprovalMetricsPublisher` Lambda emits CloudWatch metrics via the **EMF (Embedded Metric Format)** pattern: instead of calling `PutMetricData` directly, it logs structured JSON that CloudWatch Logs auto-extracts into metrics. EMF has its own per-account throttle ceiling — **100 EMF metric writes per second per account**. Cross that ceiling and CloudWatch silently drops metrics with no error visible to the Lambda.

The publisher already self-rate-limits and emits an internal `MetricEmitSkipped` count when it self-limits, but **there's no alarm on `MetricEmitSkipped > 0`**. So if approvals scale to where the publisher hits the ceiling, the dashboard quietly underreports without anyone noticing.

This pairs with [issue #4 (DLQ alarms)](./04-fanout-publisher-dlq-alarms.md): both are about "the metrics path is broken but the dashboard still shows old data." Filing as separate issues since the threshold-tuning conversation will be different (DLQ count = 1 is the right alarm; EMF skipped count needs more thought — burst spikes can be normal).

**User-visible impact:**

- Approval-volume metrics under-report at high load. Operators see lower `ApprovalRequestCount` than reality.
- No signal to "scale up the Lambda's batch size" or "adjust EMF emission cadence."
- Capacity-planning conversations rely on CloudWatch numbers that are silently lower than actual.

## Technical context

**Where the metric is already emitted:**

- `cdk/src/handlers/approval-metrics-publisher.ts` — search for `MetricEmitSkipped` (or similar; the exact name needs verification). The publisher's self-rate-limit logic increments this when it skips an EMF write to stay under the ceiling.

**What's missing:**

- `cloudwatch.Alarm` on `MetricEmitSkipped > ` over a sensible window.
- Threshold tuning: a low threshold (1 over 5 min) will fire on legitimate burst spikes; a high threshold (>100 over 1 hour) misses sustained problems. **Recommend starting at "10 over a 15-minute window"** as a conservative baseline; tune after observing real traffic.

**Why EMF instead of PutMetricData:**

- EMF batches 100x cheaper than PutMetricData for the same observability.
- The trade-off is the global per-account ceiling. PutMetricData has its own throttle but it's per-region per-account at a much higher number.
- The publisher's choice of EMF is the right one; this issue isn't asking to change that.

## Proposed fix

Add the alarm in `cdk/src/constructs/approval-metrics-publisher-consumer.ts`:

```typescript
new cloudwatch.Alarm(this, 'MetricEmitSkippedAlarm', {
metric: new cloudwatch.Metric({
namespace: 'ABCA/Cedar-HITL',
metricName: 'MetricEmitSkipped',
period: Duration.minutes(15),
statistic: 'Sum',
}),
threshold: 10,
evaluationPeriods: 1,
comparisonOperator: ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
alarmDescription: 'ApprovalMetricsPublisher hit the EMF rate ceiling and dropped metrics; consider increasing batch size or reducing per-event metric volume',
treatMissingData: TreatMissingData.NOT_BREACHING,
});
```

## Acceptance criteria

- [ ] Alarm exists on `MetricEmitSkipped` (or whatever the existing skipped-count metric is named — verify the name in `approval-metrics-publisher.ts` first)
- [ ] Threshold is documented in the alarm description with rationale
- [ ] Construct test verifies alarm presence
- [ ] If a runbook exists for "publisher hit EMF ceiling," `alarmDescription` links to it

## Out of scope

- Refactoring the publisher to use PutMetricData (different cost/perf profile, separate decision).
- Adaptive batch-size tuning (publisher could observe its own skip rate and adjust; that's a separate enhancement).
- Multi-region failover for metric publishing (out of scope for ABCA today).

## References

- `cdk/src/handlers/approval-metrics-publisher.ts`
- `cdk/src/constructs/approval-metrics-publisher-consumer.ts`
- AWS EMF reference:
- Per-account EMF ceiling:
- Companion issue: #4 (DLQ alarms, same observability gap shape)

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia leggendo cdk/src/handlers/approval-metrics-publisher.ts e verifica il nome esatto della metrica degli elementi ignorati, quindi esamina cdk/src/constructs/approval-metrics-publisher-consumer.ts per individuare gli allarmi CloudWatch e i test del construct esistenti. Aggiungi l’allarme con una soglia documentata e verifica la sua presenza nel test del construct; collega un runbook solo se ne esiste uno.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
aws, typescript
Ambito
infrastructure, observability
Tipo di issue
Funzionalità
Difficoltà
2/5
Tempo stimato
1-3 ore
Stato di attività
Tranquilla
Chiarezza
Specificata chiaramente
Idoneità per principianti
64/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.