ccl/sqlproxyccl: TestCancelQuery failed

ccl/sqlproxyccl.TestCancelQuery [failed](https://mesolite.cluster.engflow.com/invocations/default/f0b7b2a5-a201-40b7-b48c-e84a188b9c98?testReportRun=3&testReportShard=7&testReportAttempt=1#targets-Ly9wa2cvY2NsL3NxbHByb3h5Y2NsOnNxbHByb3h5Y2NsX3Rlc3Q=) on release-26.1.6-rc @ [1c1882dce8a769c6575d5d3cfb09dd12126c2f98](https://github.com/cockroachdb/cockroach/commits/1c1882dce8a769c6575d5d3cfb09dd12126c2f98):

```
pkg/util/stop/stopper.go:368 +0x36e
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer.(*Balancer).processQueue.func1()
pkg/ccl/sqlproxyccl/balancer/balancer.go:366 +0x236
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer.(*Balancer).processQueue()
pkg/ccl/sqlproxyccl/balancer/balancer.go:389 +0x1a
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer.(*Balancer).processQueue-fm()
:1 +0x47
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1()
pkg/util/stop/stopper.go:447 +0x111
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.gowrap1()
pkg/util/stop/stopper.go:448 +0x4f

Goroutine 4846 (finished) created at:
testing.(*T).Run()
GOROOT/src/testing/testing.go:1997 +0x9d2
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.TestCancelQuery()
pkg/ccl/sqlproxyccl/proxy_handler_test.go:1729 +0x1c66
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartTenant()
pkg/testutils/serverutils/test_server_shim.go:487 +0x288
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.startTestTenantPodsWithStopper()
pkg/ccl/sqlproxyccl/proxy_handler_test.go:3121 +0x196
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.startTestTenantPods()
pkg/ccl/sqlproxyccl/proxy_handler_test.go:3103 +0x5ee
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.TestCancelQuery()
pkg/ccl/sqlproxyccl/proxy_handler_test.go:1601 +0x592
github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart()
pkg/server/server_sql.go:1583 +0xb8d
github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart()
pkg/server/server.go:2141 +0x5844
github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart()
pkg/server/server.go:1732 +0x22cf
github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart()
pkg/server/testserver.go:858 +0xdd
github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start()
pkg/server/testserver.go:936 +0xc4
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start()
bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:14 +0x71
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerOnlyE()
pkg/testutils/serverutils/test_server_shim.go:345 +0x2a1
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerOnly()
pkg/testutils/serverutils/test_server_shim.go:360 +0x84
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
pkg/testutils/serverutils/test_server_shim.go:396 +0x64
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.TestCancelQuery()
pkg/ccl/sqlproxyccl/proxy_handler_test.go:1582 +0x18c
testing.tRunner()
GOROOT/src/testing/testing.go:1934 +0x21c
testing.(*T).Run.gowrap1()
GOROOT/src/testing/testing.go:1997 +0x44
==================
```

Parameters:
- attempt=1
- race=true
- run=3
- shard=7
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #169965 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-cloud-platform branch-release-25.4]
- #169046 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-cloud-platform branch-release-25.2.18-rc]
- #167577 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-cloud-platform branch-release-24.3]
- #166665 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-cloud-platform branch-release-24.3.30-rc]
- #165202 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-cloud-platform branch-release-26.1]
- #165007 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-cloud-platform branch-release-24.3.29-rc]

/cc @cockroachlabs/sqlproxy-prs

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestCancelQuery.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-65189

Start with `pkg/ccl/sqlproxyccl/proxy_handler_test.go`, especially `TestCancelQuery` around line 1729, and inspect the race-enabled failure report and stack trace. Then read the balancer queue processing in `pkg/ccl/sqlproxyccl/balancer/balancer.go` around lines 366–389. Done means identifying and fixing the cause so the test passes reliably with the race detector; the report does not specify a root cause.

Written by the indexing model from the issue text.