scylladb / scylladb/scylla-cdc-java

Add global concurrency limiter for CDC queries across all workers

Open
#170 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
32
Forks
23
Avg merge
2d 6h
Merged PRs (30d)
3

Description

Problem

When a CDC consumer starts up (especially for the first time), all workers begin querying the cluster simultaneously with no global concurrency control. The current execution model uses CompletableFuture chaining on a ScheduledExecutorService, meaning a single thread can have many outstanding async requests. The executor pool size (default: availableProcessors - 1) limits thread count but not in-flight queries.

On a cluster with 256 vnodes, this can result in 250+ concurrent CDC log readers overwhelming the cluster with queries.

Existing mechanisms (insufficient)
  • WorkerConfiguration.withMinimalWaitForWindowMs() — adds a static delay per worker but doesn't limit aggregate concurrency
  • Connection pool limits (poolingMaxRequestsPerConnectionLocal, poolingMaxQueueSize) — act as backstops but allow far too many concurrent requests in aggregate
  • ExponentialRetryBackoffWithJitter — only triggers on errors, by which point the cluster is already stressed

Proposal

Add a shared concurrency limiter (e.g., a Semaphore) across all workers to cap the number of concurrent in-flight CDC queries.

Suggested approach
  • Add a configurable max concurrent queries option to WorkerConfiguration.Builder (e.g., withMaxConcurrentQueries(int max))
  • Implement via a shared Semaphore that workers acquire before issuing a CDC query in TaskAction and release on completion
  • A sensible default (e.g., 32–64) would prevent cluster overload while still allowing reasonable throughput
  • The semaphore should be shared across all workers within a single CDCConsumer instance
Key files
  • scylla-cdc-base/src/main/java/com/scylladb/cdc/model/worker/WorkerConfiguration.java — add config option
  • scylla-cdc-base/src/main/java/com/scylladb/cdc/model/worker/TaskAction.java — add semaphore gating around query execution
  • scylla-cdc-lib/src/main/java/com/scylladb/cdc/lib/CDCConsumer.java — expose in the public builder API

Related

  • scylladb/scylla-cdc-source-connector#235 — connector-side issue for exposing this as a connector config property

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading WorkerConfiguration.java and TaskAction.java to trace how workers issue and complete CDC queries, then inspect CDCConsumer.java for the public builder flow. Define how one limiter is shared across a CDCConsumer instance, how configuration and defaults are exposed, and verify that permits cover every in-flight query and are released on completion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.