redpanda-data / redpanda-data/connect
DynamoDB writer should limit write batches to 25 items
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
DynamoDB batch writes are limited to 25 items, as per the documentation here. The benthos dynamo writer currently falls back to one-by-one writes in the case of any errors (such as exceeding this 25 item limit), which has a huge impact on throughput in the case of large output batch sizes.
It would be nice if the dynamo writer would internally limit each batch call to 25 items, making this invisible to the user and avoiding the need to restrict output batch settings to 25 or fewer in benthos config.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the DynamoDB writer and its BatchWriteItem call. Verify how output batches larger than 25 items are handled, then run the writer's relevant tests to confirm requests are split into groups of at most 25 without falling back to one-by-one writes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- database
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100