cockroachdb / cockroachdb/cockroach

changefeedccl: Improve shutdown responsiveness.

Open
#114,176 1 comment 0 reactions 0 assignees View on GitHub
A-cdc C-enhancement O-23.2-scale-testing O-testcluster T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Changefeeds are not very responsible to node shutdown
when running large scale changefeeds.

The scale in question here is: 2.5M ranges under changefeed.
See https://cockroachlabs.slack.com/archives/C05VAJ5H3QS/p1699547742213809?thread_ts=1699481426.256929&cid=C05VAJ5H3QS (log file)

The `reconcileJobStateWithLocalState` method reconstructs frontier information for that many ranges.
Critically though, this method does not check for context cancellation nor does it check
for the coordinator node being drained while it's running loops to reconstruct frontier.
This step appears to have taken in excess of 15 minutes (likely due to the fact that current
implementation of span frontier, using left leaning red-black trees, is incredibly inefficient
when it comes to memory allocation behavior, thus resulting in likely significant GOGC pressure).

Jira issue: CRDB-33382

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.