cockroachdb / cockroachdb/cockroach

basalt-26.3.2-rc-20260913: sql/stats: TestCacheAutoRefresh failed

Open
#175,419 2 comments 0 reactions 0 assignees View on GitHub
branch-basalt-26.3.2-rc-20260913 C-test-failure O-robot T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

sql/stats.TestCacheAutoRefresh [failed](https://mesolite.cluster.engflow.com/invocations/default/f9fa1f2b-bb9a-4046-8a3d-b7df01d8103c?testReportRun=1&testReportShard=8&testReportAttempt=1#targets-Ly9wa2cvc3FsL3N0YXRzOnN0YXRzX3Rlc3Q=) on basalt-26.3.2-rc-20260913 @ [c6a822fb5ec0b622c713fccacb82159d1b60bbcd](https://github.com/cockroachdb/cockroach/commits/c6a822fb5ec0b622c713fccacb82159d1b60bbcd):

```
github.com/cockroachdb/cockroach/pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4({0xa4dd768, 0xc0180c3540}, {0x9f967a0, 0xc049be02c0}, 0xc01a1fa840, 0xc01a1fa8c0)
pkg/rpc/metrics.go:589 +0x3a2
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func5({0xa4dd768, 0xc0180c3540}, {0x9f967a0, 0xc049be02c0})
pkg/rpc/context.go:918 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0xa4dd768, 0xc0180c3540})
pkg/rpc/context.go:139 +0x6b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00383dc20, {0xa4dd768, 0xc0180c3540}, {0xa0e31ee, 0x21}, 0xc004ba17c8)
pkg/util/stop/stopper.go:358 +0x237
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0xa4dd768, 0xc0180c3540}, {0x9f967a0, 0xc049be02c0}, 0xc01a1fa840, 0xc01a1fa8e0)
pkg/rpc/context.go:137 +0x14c
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func5({0xa4dd768, 0xc0180c3540}, {0x9f967a0, 0xc049be02c0})
pkg/rpc/context.go:918 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2({0xa4dd768, 0xc0180c3540}, {0x3?, 0x2?}, {0x9f967a0, 0xc049be02c0}, {0x9e1c880, 0xc028c25400}, 0x20?, {0x0, ...})
pkg/rpc/context.go:817 +0x76
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func13({0xa4dd768, 0xc0180c3540}, {0xa0e31ee, 0x21}, {0x9f967a0, 0xc049be02c0}, {0x9e1c880, 0xc028c25400}, 0x0, 0xc0190d1090, ...)
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:185 +0x4fd
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.getChainUnaryInvoker.func4({0xa4dd768, 0xc0180c3540}, {0xa0e31ee, 0x21}, {0x9f967a0, 0xc049be02c0}, {0x9e1c880, 0xc028c25400}, 0x0, {0x0, ...})
pkg/rpc/context.go:1002 +0x18d
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3({0xa4dd768, 0xc0180c34a0}, 0xc049be0160)
pkg/rpc/context.go:887 +0x467
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
pkg/rpc/context.go:1010
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc029df3080, {0xa4dd768, 0xc0180c34a0}, 0x4e7c11?, {0xa4a5c58, 0xc01ddecd80}, 0xc049be0160)
pkg/kv/kvclient/kvcoord/transport.go:229 +0x3ac
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc029df3080, {0xa4dd768, 0xc0180c34a0}, 0xc049be0160)
pkg/kv/kvclient/kvcoord/transport.go:191 +0x135
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext({{0xa5322c8?, 0xc029df3080?}}, {0xa4dd768, 0xc0180c34a0}, 0xc049be0160)
pkg/kv/kvclient/kvcoord/transport_race.go:70 +0x2df
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc003d46008, {0xa4dd768, 0xc0180c34a0}, 0xc049be0000, {0xc003d30700, 0xc005608300}, 0x0)
pkg/kv/kvclient/kvcoord/dist_sender.go:2923 +0x2a2e
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc003d46008, {0xa4dd768, 0xc0180c34a0}, 0xc049be0000, {{0xc01cb839a0, 0x4a, 0x4a}, {0xc01db4ce10, 0x4b, 0x4b}}, ...)
pkg/kv/kvclient/kvcoord/dist_sender.go:2333 +0x925
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc003d46008, {0xa4dd768, 0xc0180c34a0}, 0xc049be0000, {{0xc01cb839a0, 0x4a, 0x4a}, {0xc01db4ce10, 0x4b, 0x4b}}, ...)
pkg/kv/kvclient/kvcoord/dist_sender.go:1879 +0xf87
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc003d46008, {0xa4dbd78, 0xc046dd84b0}, 0xc049be0000)
pkg/kv/kvclient/kvcoord/dist_sender.go:1384 +0x80e
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked(0xc044516688, {0xa4dbd78, 0xc046dd84b0}, 0xc049be0000)
pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:77 +0x259
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).heartbeatLocked(0xc0445161f0, {0xa4dbd78, 0xc046dd84b0})
pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:487 +0x66b
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).heartbeat(0xc0445161f0, {0xa4dbd78, 0xc046dd84b0})
pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:443 +0x153
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).heartbeatLoop(0xc0445161f0, {0xa4dbd78, 0xc046dd84b0})
pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:413 +0x2b9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask(0xc00383dc20, {0xa4dbd78, 0xc046dd84b0}, {0xa0e5bf4, 0x21}, 0xc0384c5f88)
pkg/util/stop/stopper.go:318 +0x237
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).startHeartbeatLoopLocked.func1()
pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:366 +0x208
created by time.goFunc
GOROOT/src/time/sleep.go:215 +0x45
```

Parameters:
- attempt=1
- race=true
- run=1
- shard=8
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #174912 basalt-26.3.2-rc-20260908: sql/stats: TestCacheAutoRefresh failed [C-test-failure O-robot T-sql-queries branch-basalt-26.3.2-rc-20260908]
- #174859 sql/stats: TestCacheAutoRefresh failed [OOM] [C-test-failure O-robot T-sql-queries branch-master]

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestCacheAutoRefresh.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Contributor guide

Open the contributing guide

Research direction

Start with sql/stats.TestCacheAutoRefresh using the linked test report and the race=true parameters. Trace the reported path through pkg/rpc/metrics.go:589, pkg/rpc/context.go, and the transaction heartbeat code in pkg/kv/kvclient/kvcoord. Done means reproducing the failure, identifying its cause, and making the test pass reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.