✨ feat(aggregator): surface write-sharding events as CloudWatch metrics
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 104
Description
Problem or Use Case
Write sharding (GHSA-76rv, ADR-133) has operationally significant moments that are currently invisible to operators:
- Request exceeds shard share. A request larger than a shard's effective share (
capacity // shard_count) is rejected on every shard and can never be admitted while thatshard_countholds. The caller only seesRateLimitExceeded. - Shard count doubled. A
shard_countdoubling, whether a client-side bump onwcuexhaustion or the aggregator's proactive doubling. - Shard cap reached. A bump refused because the shard-count cap (
MAX_SHARD_COUNT) was reached.
Operators need these as signals they can alarm on, not log lines.
Proposed Solution
Emit CloudWatch metrics (and/or an audit event) for each transition:
| Metric | Dimensions | Value |
|---|---|---|
ShardCountDoubled |
namespace, resource | new shard_count |
ShardCapReached |
namespace, resource | 1 |
RequestExceedsShardShare |
namespace, resource, limit | 1 |
Emission points:
- Aggregator — where the stream shows the transition (
shard_countchange on aMODIFYrecord, proactive doubling, cap refusal). - Client-side paths — via the existing audit-event mechanism where the aggregator cannot see the event. The per-share rejection happens client-side and never produces a stream record.
Add alarm guidance to docs/monitoring.md.
Design questions:
- Should client-side emission be audit-event-only, with the aggregator turning audit events into metrics? App roles have no CloudWatch permission under the two-tier IAM model (ADR-117), so direct
PutMetricDatafrom the client is likely out. - Metric cardinality: per-entity dimensions are out; confirm namespace + resource (+ limit) is the ceiling.
Acceptance Criteria
- A metric is emitted for each of the three events (
ShardCountDoubled,ShardCapReached,RequestExceedsShardShare), with a unit test per emitter - LocalStack e2e test asserts at least one metric on a forced shard doubling
-
docs/monitoring.mdlists the three metrics and a suggested alarm - No metric carries a per-entity dimension
Related
- ADR-133, GHSA-76rv-2r9v-c5m6
- #439 (PR #466), #468, #474
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading ADR-133 and ADR-117, then trace the aggregator stream transitions and existing client-side audit-event mechanism. Run the relevant LocalStack e2e tests and inspect docs/monitoring.md. Done means all three events emit metrics with unit coverage, forced doubling is observed in e2e, monitoring guidance is documented, and no metric uses a per-entity dimension.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, cloud, documentation, observability, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100