cockroachdb / cockroachdb/cockroach

changefeedccl: webhook sink + http/2 + L4 NLB results in stream fixation

Open
#119,156 3 comments 0 reactions 0 assignees View on GitHub
A-cdc C-bug P-3 T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The combination of http/2 and Layer-4 Network Load Balancers is, in general, problematic because the NLB is unable to demultiplex the requests carried within a single TCP stream. This results in a case of "stream fixation" where a cockroach node will hammer a single backend.

This is partly a documentation issue, where customers that want to use the webhook sink with an http/2 backend must be aware that they should use a Layer-7 Application Load Balancer that is h2-clean.

From an ergonomics and troubleshooting perspective, it would be useful to have a toggle to force http/1.1 and/or a way to ensure a minimum number of TCP streams are used for h2.

Here are some redacted customer screenshots showing a large CRDB cluster with a high-volume changefeed workload sending webhook traffic to a pool of 24 backends. In the before/after comparison, one can observe a much more even distribution of HTTP requests when switching from an L4 NLB to an L7 ALB.

This shows the HTTP payload bytes per second, measured from the backend, before the investigation. Even though there are 24 other backends, only one of them is receiving meaningful traffic. Bouncing the hot node just steers the giant space laser to another backend.

image (1)

After pausing and restarting the changefeed, we see the operating regime on the left side, where HTTP payload bytes are not well distributed. During the gap, we replaced the L4 NLB with an L7 ALB that can demultiplex the h2 transport stream and distribute the workload evenly.

image

Jira issue: CRDB-36060

Epic CRDB-37341

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.