cockroachdb / cockroachdb/cockroach
changefeedccl: Improve shutdown responsiveness.
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Changefeeds are not very responsible to node shutdown
when running large scale changefeeds.
The scale in question here is: 2.5M ranges under changefeed.
See https://cockroachlabs.slack.com/archives/C05VAJ5H3QS/p1699547742213809?thread_ts=1699481426.256929&cid=C05VAJ5H3QS (log file)
The `reconcileJobStateWithLocalState` method reconstructs frontier information for that many ranges.
Critically though, this method does not check for context cancellation nor does it check
for the coordinator node being drained while it's running loops to reconstruct frontier.
This step appears to have taken in excess of 15 minutes (likely due to the fact that current
implementation of span frontier, using left leaning red-black trees, is incredibly inefficient
when it comes to memory allocation behavior, thus resulting in likely significant GOGC pressure).
Jira issue: CRDB-33382
Contributor guide
Assessment
This issue has not been assessed yet.