cockroachdb / cockroachdb/cockroach

server: gRPC server does not set NextProtocols

Open
#136,367 1 comment 0 reactions 0 assignees View on GitHub
branch-master C-bug T-db-server
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

While upgrading gRPC from v1.56.3 to v1.68.0, I noticed that post-bump CRDB was unable to connect to pre-bump CRDB nodes, the nodes using gRPC at v1.68.0 would print

> W241128 13:28:08.615942 205 server/init.go:402 ⋮ [T1,Vsystem,n?] 25 outgoing join rpc to ‹127.0.0.1:29000› unsuccessful: ‹rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: credentials: cannot check peer: missing selected ALPN property"›

This check is new in v1.68.0 (as in, it's not yet there in v1.56.3). So this problem likely persists, but it has not caused any issues. This new check was added to provide a better failure mode. Apparently every HTTP2 server needs to support ALPN, but our HTTP2 server is gRPC and it is a bit of a mystery how we end up failing this check, especially given that everything works once we disable the check, which we can do via an env var.

See the full initial analysis here: https://github.com/cockroachdb/cockroach/pull/136278#issuecomment-2506167509

With a fork of grpc-go that turns the check off, https://github.com/cockroachdb/cockroach/pull/136278 can merge. This issue serves as a reminder to get to the bottom of this problem, as future versions of gRPC are likely to hard-code the check, and we don't want to have to be on a fork forever.

**To Reproduce**

See https://github.com/cockroachdb/cockroach/pull/136278#issuecomment-2506167509

Jira issue: CRDB-45001

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.