opensearch-project / opensearch-project/data-prepper
Make DynamoDB stream shard acquisition backoff ceiling configurable
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
#7000 made shard discovery configurable via shard_discovery_interval, which removed the discovery term from shard-rotation latency. A second hardcoded interval is now the limiting factor.
StreamScheduler acquires stream partitions in a loop, sleeping between attempts on backoffCalculator.calculateBackoffToAcquireNextShard(...). In BackoffCalculator, when no partition is available the sleep grows exponentially from STARTING_BACKOFF (500ms) at BACKOFF_RATE = 2, clamped by MAX_BACKOFF_NO_SHARDS_ACQUIRED = Duration.ofSeconds(15) plus ±2s jitter. The resulting ladder is 500ms → 1s → 2s → 4s → 8s → 15s, and noAvailableShardsCount only resets when a partition is actually acquired.
Because DynamoDB rotates each stream shard roughly every 4 hours, a pipeline with a stable shard set saturates that ladder after ~6 consecutive empty attempts (~16s) and then stays pinned at the 15s ceiling. When a rotation creates a child shard, LeaderScheduler writes its partition within shard_discovery_interval (~1s when configured low), but no worker claims it until its next poll — up to 15s later. MAX_BACKOFF_NO_SHARDS_ACQUIRED is therefore now the floor on shard-rotation latency.
The symptom is the same signature described in #7000: EndToEndLatency spikes while PipelineLatency stays flat. Records are now discovered quickly, just claimed late.
Describe the solution you'd like
Add a configurable ceiling under the stream config — for example shard_acquisition_backoff_max — defaulting to the current Duration.ofSeconds(15) so there is no behavior change unless it is set, following the same validation shape as shard_discovery_interval (@NotNull plus a @DurationMin).
stream:
start_position: LATEST
shard_discovery_interval: 1s # default: 1m
shard_acquisition_backoff_max: 2s # default: 15s
Additional context
#7000 addressed the discovery term of shard-rotation latency; this issue addresses the acquisition term. Both are needed to bring rotation latency down.
#5208 (checkpoint shards and rotate through them) is a separate concern — container-count vs shard-count economics — and would, if anything, increase how often this handover path is exercised.
This is independent of the acknowledgments setting; the acquisition loop backs off identically either way.
A possible alternative worth considering: make acquisition event-driven, so LeaderScheduler resets or signals waiting workers when it creates a new partition. That would reduce rotation latency without increasing steady state coordination-store load at all, but it is a larger change than a configurable ceiling.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing StreamScheduler's acquisition loop into BackoffCalculator.calculateBackoffToAcquireNextShard and locate the stream configuration handling shard_discovery_interval. Add the shard_acquisition_backoff_max setting with the described validation and current 15-second default, then verify that the configured ceiling changes acquisition delay without changing default behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100