opensearch-project / opensearch-project/data-prepper

Make DynamoDB stream shard acquisition backoff ceiling configurable

Open
#7,124 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

untriaged
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.
#7000 made shard discovery configurable via shard_discovery_interval, which removed the discovery term from shard-rotation latency. A second hardcoded interval is now the limiting factor.

StreamScheduler acquires stream partitions in a loop, sleeping between attempts on backoffCalculator.calculateBackoffToAcquireNextShard(...). In BackoffCalculator, when no partition is available the sleep grows exponentially from STARTING_BACKOFF (500ms) at BACKOFF_RATE = 2, clamped by MAX_BACKOFF_NO_SHARDS_ACQUIRED = Duration.ofSeconds(15) plus ±2s jitter. The resulting ladder is 500ms → 1s → 2s → 4s → 8s → 15s, and noAvailableShardsCount only resets when a partition is actually acquired.

Because DynamoDB rotates each stream shard roughly every 4 hours, a pipeline with a stable shard set saturates that ladder after ~6 consecutive empty attempts (~16s) and then stays pinned at the 15s ceiling. When a rotation creates a child shard, LeaderScheduler writes its partition within shard_discovery_interval (~1s when configured low), but no worker claims it until its next poll — up to 15s later. MAX_BACKOFF_NO_SHARDS_ACQUIRED is therefore now the floor on shard-rotation latency.

The symptom is the same signature described in #7000: EndToEndLatency spikes while PipelineLatency stays flat. Records are now discovered quickly, just claimed late.

Describe the solution you'd like
Add a configurable ceiling under the stream config — for example shard_acquisition_backoff_max — defaulting to the current Duration.ofSeconds(15) so there is no behavior change unless it is set, following the same validation shape as shard_discovery_interval (@NotNull plus a @DurationMin).

stream:
    start_position: LATEST
    shard_discovery_interval: 1s          # default: 1m
    shard_acquisition_backoff_max: 2s     # default: 15s

Additional context
#7000 addressed the discovery term of shard-rotation latency; this issue addresses the acquisition term. Both are needed to bring rotation latency down.

#5208 (checkpoint shards and rotate through them) is a separate concern — container-count vs shard-count economics — and would, if anything, increase how often this handover path is exercised.

This is independent of the acknowledgments setting; the acquisition loop backs off identically either way.

A possible alternative worth considering: make acquisition event-driven, so LeaderScheduler resets or signals waiting workers when it creates a new partition. That would reduce rotation latency without increasing steady state coordination-store load at all, but it is a larger change than a configurable ceiling.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing StreamScheduler's acquisition loop into BackoffCalculator.calculateBackoffToAcquireNextShard and locate the stream configuration handling shard_discovery_interval. Add the shard_acquisition_backoff_max setting with the described validation and current 15-second default, then verify that the configured ceiling changes acquisition delay without changing default behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, databases
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.