cockroachdb / cockroachdb/cockroach
changefeedccl: webhook sink + http/2 + L4 NLB results in stream fixation
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The combination of http/2 and Layer-4 Network Load Balancers is, in general, problematic because the NLB is unable to demultiplex the requests carried within a single TCP stream. This results in a case of "stream fixation" where a cockroach node will hammer a single backend.
This is partly a documentation issue, where customers that want to use the webhook sink with an http/2 backend must be aware that they should use a Layer-7 Application Load Balancer that is h2-clean.
From an ergonomics and troubleshooting perspective, it would be useful to have a toggle to force http/1.1 and/or a way to ensure a minimum number of TCP streams are used for h2.
Here are some redacted customer screenshots showing a large CRDB cluster with a high-volume changefeed workload sending webhook traffic to a pool of 24 backends. In the before/after comparison, one can observe a much more even distribution of HTTP requests when switching from an L4 NLB to an L7 ALB.
This shows the HTTP payload bytes per second, measured from the backend, before the investigation. Even though there are 24 other backends, only one of them is receiving meaningful traffic. Bouncing the hot node just steers the giant space laser to another backend.
After pausing and restarting the changefeed, we see the operating regime on the left side, where HTTP payload bytes are not well distributed. During the gap, we replaced the L4 NLB with an L7 ALB that can demultiplex the h2 transport stream and distribute the workload evenly.
Jira issue: CRDB-36060
Epic CRDB-37341
Contributor guide
Assessment
This issue has not been assessed yet.