zeroae / zeroae/zae-limiter

✨ feat(aggregator): surface write-sharding events as CloudWatch metrics

Open
#475 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/aggregator area/limiter
Dominant language
Python
Stars
0
Forks
0
Avg merge
6h 51m
Merged PRs (30d)
104

Description

Problem or Use Case

Write sharding (GHSA-76rv, ADR-133) has operationally significant moments that are currently invisible to operators:

  1. Request exceeds shard share. A request larger than a shard's effective share (capacity // shard_count) is rejected on every shard and can never be admitted while that shard_count holds. The caller only sees RateLimitExceeded.
  2. Shard count doubled. A shard_count doubling, whether a client-side bump on wcu exhaustion or the aggregator's proactive doubling.
  3. Shard cap reached. A bump refused because the shard-count cap (MAX_SHARD_COUNT) was reached.

Operators need these as signals they can alarm on, not log lines.

Proposed Solution

Emit CloudWatch metrics (and/or an audit event) for each transition:

Metric Dimensions Value
ShardCountDoubled namespace, resource new shard_count
ShardCapReached namespace, resource 1
RequestExceedsShardShare namespace, resource, limit 1

Emission points:

  • Aggregator — where the stream shows the transition (shard_count change on a MODIFY record, proactive doubling, cap refusal).
  • Client-side paths — via the existing audit-event mechanism where the aggregator cannot see the event. The per-share rejection happens client-side and never produces a stream record.

Add alarm guidance to docs/monitoring.md.

Design questions:

  • Should client-side emission be audit-event-only, with the aggregator turning audit events into metrics? App roles have no CloudWatch permission under the two-tier IAM model (ADR-117), so direct PutMetricData from the client is likely out.
  • Metric cardinality: per-entity dimensions are out; confirm namespace + resource (+ limit) is the ceiling.

Acceptance Criteria

  • A metric is emitted for each of the three events (ShardCountDoubled, ShardCapReached, RequestExceedsShardShare), with a unit test per emitter
  • LocalStack e2e test asserts at least one metric on a forced shard doubling
  • docs/monitoring.md lists the three metrics and a suggested alarm
  • No metric carries a per-entity dimension

Related

  • ADR-133, GHSA-76rv-2r9v-c5m6
  • #439 (PR #466), #468, #474

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading ADR-133 and ADR-117, then trace the aggregator stream transitions and existing client-side audit-event mechanism. Run the relevant LocalStack e2e tests and inspect docs/monitoring.md. Done means all three events emit metrics with unit coverage, forced doubling is observed in e2e, monitoring guidance is documented, and no metric uses a per-entity dimension.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, cloud, documentation, observability, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.