Add Amazon DynamoDB sink connector
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 432
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 173
Description
### Description
Add a native Rust sink connector for Amazon DynamoDB, allowing Iggy stream messages to be written directly into DynamoDB tables.
This is tracked in the connector ecosystem roadmap: #2753
### Affected area / component
Connectors
### Proposed solution
**Crate:** [`aws-sdk-dynamodb`](https://crates.io/crates/aws-sdk-dynamodb)
(official AWS SDK, async/Tokio compatible)
**Key design decisions:**
1. **Batching** — `batch_write_item` has a hard limit of 25 items per request. The `consume()` handler will split incoming message batches into chunks of ≤25 before sending.
2. **Idempotency** — The connector will support a configurable `partition_key_field` (and optional `sort_key_field`) mapped from message payload. `batch_write_item` uses `PutRequest` which overwrites
existing items on key conflict, making repeated delivery idempotent as long as the primary key is deterministic.
3. **UnprocessedItems handling** — `batch_write_item` can return partial failures via `UnprocessedItems` without throwing an error. The connector will retry unprocessed items with exponential backoff rather than silently dropping them.
4. **Throughput errors** — `ProvisionedThroughputExceededException` will be handled with retry logic compatible with both Provisioned and On-Demand capacity modes.
5. **Item size limit** — DynamoDB enforces a 400KB per-item limit. Oversized messages will be logged and skipped rather than failing the entire batch.
### Alternatives considered
_No response_
### Contribution
- [x] I'm willing to submit a pull request to implement this feature
### Good first issue
- [ ] I think this could be a good first issue for a new contributor
Contributor guide
Assessment
This issue has not been assessed yet.