redpanda-data / redpanda-data/connect

aws_dynamodb_cdc: Support namespace or client_id for checkpoint isolation

Open
#4,319 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

Context: In our development environment, multiple engineers frequently run the same Redpanda Connect pipeline against a shared DynamoDB table (CDC stream) for testing. Currently, all instances point to the same checkpoint_table.

Problem: Checkpoints in the DynamoDB table are currently keyed only by (StreamArn, ShardID). There is no concept of a "Consumer Group" or logical scope. When multiple developers run the pipeline simultaneously:

They contend for the same checkpoint rows. Sequence numbers are overwritten by different consumers. This leads to skipped events, duplicates, and inconsistent state across the dev team.

While creating separate checkpoint tables per developer is a workaround, it creates significant infrastructure sprawl and is operationally difficult to manage at scale.

Ask: Could the aws_dynamodb_cdc input be updated to support an optional namespace, consumer_group, or similar identifier?

The goal is to allow multiple independent logical readers to share a single physical checkpoint table without colliding. This would bring the DynamoDB CDC input in line with the "one store, many logical readers" pattern and significantly improve the developer experience by reducing infrastructure sprawl.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the aws_dynamodb_cdc input and the code that constructs checkpoint keys from StreamArn and ShardID. Read the surrounding configuration and tests to determine how an optional logical scope should be represented; done means independent readers can share one checkpoint table without overwriting one another, with coverage for isolated checkpoints.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
databases, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.