cockroachdb / cockroachdb/cockroach

sql/physicalplan: TestFakeSpanResolver failed [timeout]

Closed
#175,435 2 comments 0 reactions 0 assignees View on GitHub
branch-master C-test-failure O-robot T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

sql/physicalplan.TestFakeSpanResolver [failed](https://mesolite.cluster.engflow.com/invocations/default/4191afe3-a742-4f74-b800-acd2f3812927?testReportRun=1&testReportShard=3&testReportAttempt=1#targets-Ly9wa2cvc3FsL3BoeXNpY2FscGxhbjpwaHlzaWNhbHBsYW5fdGVzdA==) on master @ [a96c513d8e167b3b9311d5e414a310b622b61cd2](https://github.com/cockroachdb/cockroach/commits/a96c513d8e167b3b9311d5e414a310b622b61cd2):

```
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0xa69a750, 0xc0284e7b30})
pkg/rpc/context.go:154 +0x6b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc005f370e0, {0xa69a750, 0xc0284e7b30}, {0xa2303d4, 0x21}, 0xc00cb69598)
pkg/util/stop/stopper.go:358 +0x237
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0xa69a750, 0xc0284e7b30}, {0xa0deba0, 0xc033c79380}, 0xc001ba4b20, 0xc001ba4da0)
pkg/rpc/context.go:152 +0x14c
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func5({0xa69a750, 0xc0284e7b30}, {0xa0deba0, 0xc033c79380})
pkg/rpc/context.go:933 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2({0xa69a750, 0xc0284e7b30}, {0x4e5ff9?, 0x4e7e71?}, {0xa0deba0, 0xc033c79380}, {0x9f5ffc0, 0xc001d3a300}, 0x10001004e7ea9?, {0x0, ...})
pkg/rpc/context.go:832 +0x76
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func13({0xa69a750, 0xc0284e7b30}, {0xa2303d4, 0x21}, {0xa0deba0, 0xc033c79380}, {0x9f5ffc0, 0xc001d3a300}, 0x0, 0xc008569220, ...)
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:185 +0x4fd
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.getChainUnaryInvoker.func4({0xa69a750, 0xc0284e7b30}, {0xa2303d4, 0x21}, {0xa0deba0, 0xc033c79380}, {0x9f5ffc0, 0xc001d3a300}, 0x0, {0x0, ...})
pkg/rpc/context.go:1017 +0x18d
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3({0xa69b9e8, 0xc024488e60}, 0xc033c79200)
pkg/rpc/context.go:902 +0x471
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
pkg/rpc/context.go:1025
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc00b364a80, {0xa69b9e8, 0xc024488e60}, 0x4e7e71?, {0xa65f0d0, 0xc004c44d20}, 0xc033c79200)
pkg/kv/kvclient/kvcoord/transport.go:231 +0x48c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc00b364a80, {0xa69b9e8, 0xc024488e60}, 0xc033c79200)
pkg/kv/kvclient/kvcoord/transport.go:192 +0x135
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext({{0xa6ed290?, 0xc00b364a80?}}, {0xa69b9e8, 0xc024488e60}, 0xc033c79200)
pkg/kv/kvclient/kvcoord/transport_race.go:70 +0x2df
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc005f4fc08, {0xa69b9e8, 0xc024488e60}, 0xc033c79080, {0xc0060d70a0, 0xc00b3bda00}, 0x0)
pkg/kv/kvclient/kvcoord/dist_sender.go:2925 +0x2a5c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc005f4fc08, {0xa69b9e8, 0xc024488e60}, 0xc033c79080, {{0xc00e4ccff8, 0x4, 0x8}, {0xc00e4ccff8, 0x5, 0x8}}, ...)
pkg/kv/kvclient/kvcoord/dist_sender.go:2335 +0x925
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc005f4fc08, {0xa69b9e8, 0xc024488e60}, 0xc033c79080, {{0xc00e4ccff8, 0x4, 0x8}, {0xc00e4ccff8, 0x5, 0x8}}, ...)
pkg/kv/kvclient/kvcoord/dist_sender.go:1881 +0xf93
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc005f4fc08, {0xa69b9e8, 0xc020a5fc20}, 0xc033c79080)
pkg/kv/kvclient/kvcoord/dist_sender.go:1386 +0x826
github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send(0xc005ff3c98, {0xa69b9e8, 0xc020a5fc20}, 0xc033c79080)
pkg/kv/db.go:233 +0x222
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc005ff3c30, {0xa69b9e8, 0xc020a5fc20}, 0xc033c79080, {0xa64bd00, 0xc005ff3c98})
pkg/kv/db.go:1277 +0x17e
github.com/cockroachdb/cockroach/pkg/kv.(*DB).send(0xc005ff3c30, {0xa69b9e8, 0xc020a5fc20}, 0xc033c79080)
pkg/kv/db.go:1260 +0x189
github.com/cockroachdb/cockroach/pkg/kv.sendAndFill({0xa69b9e8, 0xc020a5fc20}, 0xc00cb6ba90, 0xc02b5c7808)
pkg/kv/db.go:1032 +0x2ba
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run(0xc005ff3c30, {0xa69b9e8, 0xc020a5fc20}, 0xc02b5c7808)
pkg/kv/db.go:1055 +0xa8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait.(*Queue).queryTxnStatus(0xc016c18cf0, {0xa69b9e8, 0xc020a5fc20}, {{0xcc, 0x6f, 0x1a, 0x2d, 0x51, 0x73, 0x43, ...}, ...}, ...)
pkg/kv/kvserver/txnwait/queue.go:1080 +0x38b
github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait.(*Queue).startQueryPusherTxn.func1({0xa69b9e8, 0xc020a5fc20})
pkg/kv/kvserver/txnwait/queue.go:986 +0x38f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0xa69b9e8, 0xc020a5fc20})
pkg/util/stop/stopper.go:447 +0x113
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 507058
pkg/util/stop/stopper.go:445 +0x247
```

Parameters:
- attempt=1
- race=true
- run=1
- shard=3
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #174929 basalt-26.3: sql/physicalplan: TestFakeSpanResolver failed [C-test-failure O-robot T-sql-queries branch-basalt-26.3]
- #174919 basalt-26.3.2-rc-20260908: sql/physicalplan: TestFakeSpanResolver failed [C-test-failure O-robot T-sql-queries branch-basalt-26.3.2-rc-20260908]
- #174877 basalt-26.3: sql/physicalplan: TestFakeSpanResolver failed [C-test-failure O-robot T-sql-queries branch-basalt-26.3]
- #174785 basalt-26.3: sql/physicalplan: TestFakeSpanResolver failed [C-test-failure O-robot T-sql-queries branch-basalt-26.3]
- #172053 release-26.3: sql/physicalplan: TestFakeSpanResolver failed [C-test-failure O-robot T-sql-queries branch-release-26.3 v26.4.0-prerelease]

[Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68370

Contributor guide

Open the contributing guide

Research direction

Start by reproducing sql/physicalplan.TestFakeSpanResolver with race=true, run=1, and shard=3, then inspect the stack trace and the related failures listed in the report. Trace the timeout through the mentioned RPC, KV, and transaction-wait paths; done means identifying the cause and making the test pass reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.