aws / aws/agent-toolkit-for-aws
aurora-dsql: Add observability (CloudWatch metrics) and CDC-to-Iceberg streaming guidance
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 306
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 51
Description
## Describe the feature
The Aurora DSQL skill currently covers schema design, authentication, query plans, DDL migrations, scaling, and MySQL migration. However, it has no guidance on two recently-documented capabilities:
1. **Observability via CloudWatch** — DSQL-specific metrics (commit latency, transaction throughput, OCC conflict rates, connection counts), dashboard setup, and alarm configuration
2. **CDC streaming to Apache Iceberg** — getting data out of DSQL for real-time analytics via change data capture
Both are covered by recent AWS blog posts but not reflected in the skill:
- [Streaming Aurora DSQL changes into Apache Iceberg](https://aws.amazon.com/blogs/database/unlocking-real-time-analytics-streaming-aurora-dsql-changes-into-apache-iceberg/)
- [Amazon Aurora DSQL observability concepts and usage with Amazon CloudWatch](https://aws.amazon.com/blogs/database/amazon-aurora-dsql-observability-concepts-and-usage-with-amazon-cloudwatch/)
### Proposed additions:
**1. `references/observability.md`** — covering:
- DSQL-specific CloudWatch metrics namespace and metric names
- Key metrics to monitor (commit latency p50/p95/p99, transaction throughput, OCC conflict rate, connection pool utilization)
- Recommended CloudWatch dashboard layout
- Alarm thresholds and escalation (conflict rate > N% → OCC retry storm likely, commit latency > Nms → check multi-region topology)
- Relationship between DSQL's OCC model and observable conflict metrics
- Troubleshooting decision tree: high latency → check conflicts vs check region topology vs check query plan
**2. `references/cdc-streaming.md`** — covering:
- Architecture: DSQL → CDC mechanism → Apache Iceberg (on S3/S3 Tables)
- Setup and prerequisites
- Schema evolution handling (DSQL DDL changes → Iceberg schema evolution)
- Latency characteristics (how fresh is the Iceberg data?)
- Cost model (what does the CDC pipeline cost per GB of changes?)
- When to use this vs direct queries on DSQL (analytics workload routing — same principle as DynamoDB's 'move analytical workloads off the operational table')
- Failure modes and monitoring (CDC lag, schema mismatch, pipeline stalls)
**3. SKILL.md reference entries** — pointing agents to both documents when the user asks about monitoring, observability, analytics, or getting data out of DSQL.
## Use Case
- "How do I know my DSQL cluster is healthy?" → currently no answer in the skill
- "How do I monitor OCC conflict rates?" → the skill mentions OCC but not how to observe it
- "I need to run analytics/BI on my DSQL data without impacting the operational workload" → currently no answer
- "How do I stream DSQL changes to a data lake?" → currently no answer
These are both operational maturity questions that arise after initial deployment — the skill covers getting started but not running in production.
## Proposed Solution
Two new reference documents + SKILL.md entries. Content should be derived from the official blog posts above plus the DSQL documentation. The observability guide should include concrete metric names and alarm configurations (not just "monitor your database"). The CDC guide should include the architecture diagram from the blog and a cost/latency model.
## Other Information
- The DynamoDB skill already has analogous content: Integration #8 (Zero-ETL routing) and the observability best practice (CloudWatch alarms on ThrottledRequests, SystemErrors). The DSQL skill should have equivalent operational guidance for its own metric surface.
- The existing `references/auth/scaling-guide.md` mentions latency and throughput in passing but provides no monitoring guidance — the observability doc would complement it.
- These blog posts were published in 2026, likely after the skill's last content update.
Contributor guide
Research direction
Read the two linked AWS blog posts, the existing SKILL.md, and references/auth/scaling-guide.md first. Add references/observability.md and references/cdc-streaming.md with the requested metrics, architecture, setup, latency, cost, schema, and failure-mode guidance, then add SKILL.md entries that direct monitoring and analytics questions to them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- data-engineering, documentation, observability
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100