aws / aws/agent-toolkit-for-aws

aurora-dsql: Add observability (CloudWatch metrics) and CDC-to-Iceberg streaming guidance

Open
#262 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.6k
Forks
306
Avg merge
1d 1h
Merged PRs (30d)
51

Description

## Describe the feature

The Aurora DSQL skill currently covers schema design, authentication, query plans, DDL migrations, scaling, and MySQL migration. However, it has no guidance on two recently-documented capabilities:

1. **Observability via CloudWatch** — DSQL-specific metrics (commit latency, transaction throughput, OCC conflict rates, connection counts), dashboard setup, and alarm configuration
2. **CDC streaming to Apache Iceberg** — getting data out of DSQL for real-time analytics via change data capture

Both are covered by recent AWS blog posts but not reflected in the skill:

- [Streaming Aurora DSQL changes into Apache Iceberg](https://aws.amazon.com/blogs/database/unlocking-real-time-analytics-streaming-aurora-dsql-changes-into-apache-iceberg/)
- [Amazon Aurora DSQL observability concepts and usage with Amazon CloudWatch](https://aws.amazon.com/blogs/database/amazon-aurora-dsql-observability-concepts-and-usage-with-amazon-cloudwatch/)

### Proposed additions:

**1. `references/observability.md`** — covering:
- DSQL-specific CloudWatch metrics namespace and metric names
- Key metrics to monitor (commit latency p50/p95/p99, transaction throughput, OCC conflict rate, connection pool utilization)
- Recommended CloudWatch dashboard layout
- Alarm thresholds and escalation (conflict rate > N% → OCC retry storm likely, commit latency > Nms → check multi-region topology)
- Relationship between DSQL's OCC model and observable conflict metrics
- Troubleshooting decision tree: high latency → check conflicts vs check region topology vs check query plan

**2. `references/cdc-streaming.md`** — covering:
- Architecture: DSQL → CDC mechanism → Apache Iceberg (on S3/S3 Tables)
- Setup and prerequisites
- Schema evolution handling (DSQL DDL changes → Iceberg schema evolution)
- Latency characteristics (how fresh is the Iceberg data?)
- Cost model (what does the CDC pipeline cost per GB of changes?)
- When to use this vs direct queries on DSQL (analytics workload routing — same principle as DynamoDB's 'move analytical workloads off the operational table')
- Failure modes and monitoring (CDC lag, schema mismatch, pipeline stalls)

**3. SKILL.md reference entries** — pointing agents to both documents when the user asks about monitoring, observability, analytics, or getting data out of DSQL.

## Use Case

- "How do I know my DSQL cluster is healthy?" → currently no answer in the skill
- "How do I monitor OCC conflict rates?" → the skill mentions OCC but not how to observe it
- "I need to run analytics/BI on my DSQL data without impacting the operational workload" → currently no answer
- "How do I stream DSQL changes to a data lake?" → currently no answer

These are both operational maturity questions that arise after initial deployment — the skill covers getting started but not running in production.

## Proposed Solution

Two new reference documents + SKILL.md entries. Content should be derived from the official blog posts above plus the DSQL documentation. The observability guide should include concrete metric names and alarm configurations (not just "monitor your database"). The CDC guide should include the architecture diagram from the blog and a cost/latency model.

## Other Information

- The DynamoDB skill already has analogous content: Integration #8 (Zero-ETL routing) and the observability best practice (CloudWatch alarms on ThrottledRequests, SystemErrors). The DSQL skill should have equivalent operational guidance for its own metric surface.
- The existing `references/auth/scaling-guide.md` mentions latency and throughput in passing but provides no monitoring guidance — the observability doc would complement it.
- These blog posts were published in 2026, likely after the skill's last content update.

Contributor guide

Open the contributing guide

Research direction

Read the two linked AWS blog posts, the existing SKILL.md, and references/auth/scaling-guide.md first. Add references/observability.md and references/cdc-streaming.md with the requested metrics, architecture, setup, latency, cost, schema, and failure-mode guidance, then add SKILL.md entries that direct monitoring and analytics questions to them.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
data-engineering, documentation, observability
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.