cockroachdb / cockroachdb/cockroach

bulkmerge: auto-tune RPC flow control window size on memory pressure during final merge iteration

Open
#164,789 0 comments 0 reactions 0 assignees View on GitHub
C-enhancement T-sql-foundations
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

## Summary

The final merge iteration of the distributed bulk merge pipeline now has memory accounting for the blob service RPC transport buffers (added in #164179). Each concurrent remote SST stream buffers up to `window × blobs.ChunkSize` bytes on the receiving side, where `window` is the flow control window size controlled by the `bulkio.blob.flow_control_window` cluster setting, introduced in #163941 for `GetStreamFlowControlled`.

Currently, all knobs that govern memory in the final merge iteration — the RPC flow control window size (`bulkio.blob.flow_control_window`) and the in-flight fraction (`bulkio.merge.rpc_inflight_fraction`) — are manually set cluster settings. When a job runs into memory pressure, there is no automatic relief: the operation either succeeds or fails with a memory error.

This issue tracks automatically tuning the RPC flow control window size downward when memory errors are encountered, transparently to the user.

## Problem

When the merge processor hits a memory limit during the final iteration, the most direct lever is the RPC flow control window size: reducing it lowers the per-stream buffer ceiling (`window × blobs.ChunkSize`), which reduces the peak memory reservation across all concurrent remote streams.

However, `bulkio.blob.flow_control_window` is currently a **cluster setting**. We cannot change it in response to a single job's memory pressure because doing so would affect every other operation on the cluster simultaneously.

## Proposed Solution

### 1. Add per-connection window size support to `GetStreamFlowControlled`

Rather than having the server-side handler read the window size exclusively from `bulkio.blob.flow_control_window`, allow the window size to be passed as a parameter in the RPC request (or as a per-stream option). The cluster setting can serve as the default, but callers that need a smaller window can override it on a per-connection basis without affecting other users.

**Relevant code:**
- [stream.go](https://github.com/cockroachdb/cockroach/blob/d3b44e6555155fc65d0dd6776b58ac16f61fe391/pkg/blobs/stream.go) — `GetStreamFlowControlled` implementation and `bulkio.blob.flow_control_window`
- [blobs.proto](https://github.com/cockroachdb/cockroach/blob/d3b44e6555155fc65d0dd6776b58ac16f61fe391/pkg/blobs/blobspb/blobs.proto) — RPC request message; the window size could be added here

### 2. Implement an auto-tuning retry loop in the merge processor

In the final merge iteration inside the merge processor, wrap the operation in a retry loop that responds to memory errors by reducing the RPC window size:

```
windowSize = defaultFromClusterSetting() // bulkio.blob.flow_control_window
for {
err = runFinalMergeIteration(windowSize)
if err == nil {
break // success
}
if !isMemoryError(err) {
return err // non-memory failure, propagate immediately
}
if windowSize <= minimumWindowSize {
return err // already at floor, nothing left to tune — fail with memory error
}
windowSize = max(windowSize / 2, minimumWindowSize)
log.Infof("memory pressure detected during final merge; retrying with window size %d", windowSize)
}
```

The minimum window size (floor) should be chosen such that it is still functionally useful (e.g., 1 chunk in-flight) while being a meaningful reduction. The number of retry attempts is bounded by the number of halvings from the default down to the minimum.

**Relevant code:**
- [merge_processor.go](https://github.com/cockroachdb/cockroach/blob/d3b44e6555155fc65d0dd6776b58ac16f61fe391/pkg/sql/bulkmerge/merge_processor.go) — final merge iteration and RPC memory accounting; `bulkio.merge.rpc_inflight_fraction` cluster setting

## Next Steps

- [ ] Extend `GetStreamFlowControlledRequest` (or equivalent) in the blobs proto to accept an optional per-connection window size override
- [ ] Update `GetStreamFlowControlled` server and client implementations to honor the per-connection override when present, falling back to `bulkio.blob.flow_control_window` when not specified
- [ ] Update the merge processor to pass the window size explicitly when opening remote SST streams
- [ ] Implement the retry loop in the merge processor's final iteration, reducing the window on memory errors
- [ ] Add a minimum window size constant / cluster setting floor
- [ ] Add a test that exercises the auto-tuning path (e.g., a test that constrains the memory budget to force at least one window reduction)

## Related

- #161865 — parent issue: reduce memory consumption during final merge iteration
- #163941 — blobs: add flow-controlled streaming RPC with sliding window
- #164179 — bulkmerge: reserve memory for RPC transport buffers during merge

Epic CRDB-62564
Jira issue: CRDB-61017

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.