cockroachdb / cockroachdb/cockroach

basalt-26.3.2-rc-20260916: testutils/testcluster: TestClusterSqlDisabled failed

Open
#175,697 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

branch-basalt-26.3.2-rc-20260916 C-test-failure O-robot release-blocker T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

testutils/testcluster.TestClusterSqlDisabled failed on basalt-26.3.2-rc-20260916 @ 024cf0f74e8abea8b9fe2f4a05441dac7dc66c41:

google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0303ddc20, {0xc0278c1b40, 0x5, 0x5})
	external/org_golang_google_grpc/internal/transport/transport.go:498 +0x11f
google.golang.org/grpc.(*parser).recvMsg(0xc0278c1b30, 0x7fffffff)
	external/org_golang_google_grpc/rpc_util.go:600 +0x6a
google.golang.org/grpc.recvAndDecompress(0xc0278c1b30, 0xc0303ddc20, {0x0, 0x0}, 0x7fffffff, 0x0, {0xa3b6830, 0x11d5c600})
	external/org_golang_google_grpc/rpc_util.go:730 +0x7c
google.golang.org/grpc.recv(0xc0278c1b30, {0x7f8fc02d2f10, 0xc0012a9e60}, 0xc0303ddc20, {0x0, 0x0}, {0x9e9ac60, 0xc003c18d80}, 0x7fffffff, 0x0, ...)
	external/org_golang_google_grpc/rpc_util.go:796 +0xc8
google.golang.org/grpc.(*serverStream).RecvMsg(0xc02f03c3c0, {0x9e9ac60, 0xc003c18d80})
	external/org_golang_google_grpc/stream.go:1712 +0x2f1
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchStreamImpl(0xc0035c8008, {0x7f8fba595100, 0xc0338d4e70})
	pkg/server/node.go:1968 +0xc2
github.com/cockroachdb/cockroach/pkg/server.(*Node).BatchStream(0xc0035c8008, {0xa42bb58, 0xc0338d4e70})
	pkg/server/node.go:1955 +0x5a
github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_BatchStream_Handler({0x9e8cee0, 0xc0035c8008}, {0xa415a78, 0xc02f03c3c0})
	bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:12768 +0xe6
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x9e8cee0, 0xc0035c8008}, {0xa415a78, 0xc02f03c3c0}, 0xc036760000, 0xa347ce0)
	pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:107 +0x710
google.golang.org/grpc.getChainStreamHandler.func1({0x9e8cee0, 0xc0035c8008}, {0xa415a78, 0xc02f03c3c0})
	external/org_golang_google_grpc/server.go:1504 +0x231
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x9e8cee0, 0xc0035c8008}, {0xa415a78, 0xc02f03c3c0}, 0xc036760000, 0xc01d0d7200)
	pkg/rpc/context.go:191 +0xd7
google.golang.org/grpc.getChainStreamHandler.func1.getChainStreamHandler.1({0x9e8cee0, 0xc0035c8008}, {0xa415a78, 0xc02f03c3c0})
	external/org_golang_google_grpc/server.go:1504 +0x116
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc0028c0000, {{0x1}, {0xa4386e0, 0xc002454a20}}, 0x0}, {0x9e8cee0, 0xc0035c8008}, {0xa415a78, 0xc02f03c3c0}, 0xc036760000, ...)
	pkg/rpc/auth.go:192 +0x57d
google.golang.org/grpc.getChainStreamHandler.func1({0x9e8cee0, 0xc0035c8008}, {0xa415a78, 0xc02f03c3c0})
	external/org_golang_google_grpc/server.go:1504 +0x231
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0xc0026ab040?, 0xa3d9490?})
	pkg/rpc/context.go:148 +0x66
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0026ab040, {0xa3d9490, 0xc0278c1b90}, {0xc00c6ddf20, 0x27}, 0xc01178f7d0)
	pkg/util/stop/stopper.go:358 +0x237
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x9e8cee0, 0xc0035c8008}, {0xa415a78, 0xc02f03c3c0}, 0xc036760000, 0xc01d0d7180)
	pkg/rpc/context.go:147 +0x15a
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x9e8cee0, 0xc0035c8008}, {0xa415a78, 0xc02f03c3c0}, 0xc036760000, 0xa347ce0)
	external/org_golang_google_grpc/server.go:1495 +0xdd
google.golang.org/grpc.(*Server).processStreamingRPC(0xc0028ef680, {0xa4399a0, 0xc00461dba0}, 0xc0303ddc20, 0xc0018366f0, 0x10783de0, 0x0)
	external/org_golang_google_grpc/server.go:1659 +0x20f5
google.golang.org/grpc.(*Server).handleStream(0xc0028ef680, {0xa4399a0, 0xc00461dba0}, 0xc0303ddc20, 0x0)
	external/org_golang_google_grpc/server.go:1739 +0xfa5
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	external/org_golang_google_grpc/server.go:970 +0x173
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 6159
	external/org_golang_google_grpc/server.go:981 +0x253

goroutine 391116 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
	external/org_golang_google_grpc/stream.go:383 +0xe7
created by google.golang.org/grpc.newClientStreamWithParams in goroutine 391115
	external/org_golang_google_grpc/stream.go:382 +0x1c9f

Parameters:

  • attempt=1
  • race=true
  • run=3
  • shard=2
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

  • #175667 release-26.4: testutils/testcluster: TestClusterSqlDisabled failed [C-test-failure O-robot T-testeng branch-release-26.4 release-blocker]
  • #175659 basalt-26.3.2-rc-20260908: testutils/testcluster: TestClusterSqlDisabled failed [C-test-failure O-robot T-testeng branch-basalt-26.3.2-rc-20260908 release-blocker]

/cc @cockroachlabs/test-eng

This test on roachdash | Improve this report!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with testutils/testcluster.TestClusterSqlDisabled and inspect the linked Mesolite invocation, then trace the reported stack through pkg/server/node.go's BatchStream entry point. Reproduce the failure with race=true and run=3, shard=2. Done means the underlying failure is addressed and the test passes on the affected branch.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.