cockroachdb / cockroachdb/cockroach

basalt-26.3.2-rc-20260916: sql/stats: TestCacheAutoRefresh failed

Open
#175,583 2 comments 0 reactions 0 assignees View on GitHub
branch-basalt-26.3.2-rc-20260916 C-test-failure O-robot release-blocker T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

sql/stats.TestCacheAutoRefresh [failed](https://mesolite.cluster.engflow.com/invocations/default/9f7a7420-cbb9-454a-8939-99ee51e4cf97?testReportRun=1&testReportShard=8&testReportAttempt=1#targets-Ly9wa2cvc3FsL3N0YXRzOnN0YXRzX3Rlc3Q=) on basalt-26.3.2-rc-20260916 @ [20b5b7762461dc7d59206ab797cd12d09c877eaa](https://github.com/cockroachdb/cockroach/commits/20b5b7762461dc7d59206ab797cd12d09c877eaa):

```
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWorkStats(0xc0037f9008, {0xa4ec6c8, 0xc0302b2640}, 0xc014dc6840, 0xc0320581c0, {0x0?})
pkg/kv/kvserver/replica_send.go:196 +0x92d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWorkStats(0xc0060e0008, {0xa4ec6c8, 0xc0302b25a8}, 0xc014dc6840, 0xc0320581c0, {0xc02a6a82b0?})
pkg/kv/kvserver/store_send.go:242 +0xc65
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWorkStats(0xc00104f0e0, {0xa4ec6c8, 0xc0302b25a8}, 0xc014dc6840, 0xc0320581c0, {0x0?})
pkg/kv/kvserver/stores.go:195 +0xfc
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc001d63008, {0xa4ec6c8, 0xc0302b25a8}, {0x3?}, 0xc014dc6840)
pkg/server/node.go:1736 +0xbc5
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc001d63008, {0xa4ead10, 0xc010f4d770}, 0xc014dc6840)
pkg/server/node.go:1929 +0x3f6
github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler.func1({0xa4ead10, 0xc010f4d770}, {0x9fa4a00, 0xc014dc6840})
bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:12762 +0xbf
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0xa4ead10, 0xc010f4d770}, {0x9fa4a00, 0xc014dc6840}, 0xc02e064f20, 0xc03850b638)
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:60 +0x74c
google.golang.org/grpc.getChainUnaryHandler.func1({0xa4ead10, 0xc010f4d770}, {0x9fa4a00, 0xc014dc6840})
external/org_golang_google_grpc/server.go:1177 +0x23d
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0xa4ead10, 0xc010f4d770}, {0x9fa4a00, 0xc014dc6840}, 0xc02e064f20, 0xc031cd75c0)
pkg/rpc/context.go:182 +0xf1
google.golang.org/grpc.getChainUnaryHandler.func1.getChainUnaryHandler.1({0xa4ead10, 0xc010f4d770}, {0x9fa4a00, 0xc014dc6840})
external/org_golang_google_grpc/server.go:1177 +0x11f
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc00211a000, {{0x1}, {0xa54a820, 0xc00230bce0}}, 0x0}, {0xa4ead10, 0xc010f4d770}, {0x9fa4a00, 0xc014dc6840}, 0xc02e064f20, ...)
pkg/rpc/auth.go:96 +0x325
google.golang.org/grpc.getChainUnaryHandler.func1({0xa4ead10, 0xc010f4d770}, {0x9fa4a00, 0xc014dc6840})
external/org_golang_google_grpc/server.go:1177 +0x23d
github.com/cockroachdb/cockroach/pkg/rpc.gatewayRequestRecoveryInterceptor({0xa4ead10, 0xc010f4d770}, {0x9fa4a00, 0xc014dc6840}, 0xc02e064f20?, 0xc031cd7540)
pkg/rpc/metrics.go:803 +0x183
google.golang.org/grpc.getChainUnaryHandler.func1.getChainUnaryHandler.1({0xa4ead10, 0xc010f4d770}, {0x9fa4a00, 0xc014dc6840})
external/org_golang_google_grpc/server.go:1177 +0x11f
github.com/cockroachdb/cockroach/pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4({0xa4ead10, 0xc010f4d770}, {0x9fa4a00, 0xc014dc6840}, 0xc02e064f20, 0xc031cd7500)
pkg/rpc/metrics.go:589 +0x3a2
google.golang.org/grpc.getChainUnaryHandler.func1({0xa4ead10, 0xc010f4d770}, {0x9fa4a00, 0xc014dc6840})
external/org_golang_google_grpc/server.go:1177 +0x23d
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0xa4ead10, 0xc010f4d770})
pkg/rpc/context.go:139 +0x6b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc001c8f5e0, {0xa4ead10, 0xc010f4d770}, {0xa0f1545, 0x21}, 0xc02a6a96c8)
pkg/util/stop/stopper.go:358 +0x237
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0xa4ead10, 0xc010f4d770}, {0x9fa4a00, 0xc014dc6840}, 0xc02e064f20, 0xc031cd74c0)
pkg/rpc/context.go:137 +0x14c
google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0xa4ead10, 0xc010f4d770}, {0x9fa4a00, 0xc014dc6840}, 0xc02e064f20, 0xc03850b638)
external/org_golang_google_grpc/server.go:1168 +0xe8
github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler({0x9f96c80, 0xc001d63008}, {0xa4ead10, 0xc010f4d770}, 0xc02ce865b0, 0xc0035d0280)
bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:12764 +0x1e7
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00247cd20, {0xa54ba80, 0xc003286000}, 0xc0161447e0, 0xc001d09b60, 0x10926700, 0x0)
external/org_golang_google_grpc/server.go:1358 +0x1871
google.golang.org/grpc.(*Server).handleStream(0xc00247cd20, {0xa54ba80, 0xc003286000}, 0xc0161447e0, 0x0)
external/org_golang_google_grpc/server.go:1735 +0xff3
google.golang.org/grpc.(*Server).serveStreams.func1.1()
external/org_golang_google_grpc/server.go:970 +0x173
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 6023
external/org_golang_google_grpc/server.go:981 +0x253
```

Parameters:
- attempt=1
- race=true
- run=1
- shard=8
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #175423 basalt-26.3: sql/stats: TestCacheAutoRefresh failed [C-test-failure O-robot T-sql-queries branch-basalt-26.3]
- #175419 basalt-26.3.2-rc-20260913: sql/stats: TestCacheAutoRefresh failed [C-test-failure O-robot T-sql-queries branch-basalt-26.3.2-rc-20260913]
- #174912 basalt-26.3.2-rc-20260908: sql/stats: TestCacheAutoRefresh failed [C-test-failure O-robot T-sql-queries branch-basalt-26.3.2-rc-20260908]

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestCacheAutoRefresh.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing sql/stats.TestCacheAutoRefresh with race detection using the linked invocation details, then inspect the reported call path beginning at pkg/kv/kvserver/replica_send.go:196 and the related store and node send functions. Done means the test no longer fails on the affected basalt branches, with the underlying failure explained and covered by the test run.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.