ccl/sqlproxyccl: TestCancelQuery failed

ccl/sqlproxyccl.TestCancelQuery [failed](https://mesolite.cluster.engflow.com/invocations/default/8a7c983d-69ac-4caa-a6ea-62825000e84c?testReportRun=2&testReportShard=7&testReportAttempt=1#targets-Ly9wa2cvY2NsL3NxbHByb3h5Y2NsOnNxbHByb3h5Y2NsX3Rlc3Q=) on staging-v25.4.16 @ [65ac8a6bd3d10290de3f25297aaa6d1c3de3c369](https://github.com/cockroachdb/cockroach/commits/65ac8a6bd3d10290de3f25297aaa6d1c3de3c369):

```
pkg/ccl/sqlproxyccl/conn_migration.go:49 +0x31
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*forwarder).TransferConnection()
pkg/ccl/sqlproxyccl/conn_migration.go:149 +0x124
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer.(*Balancer).processQueue.func1.1()
pkg/ccl/sqlproxyccl/balancer/balancer.go:374 +0x27b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1()
pkg/util/stop/stopper.go:447 +0x115
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.gowrap1()
pkg/util/stop/stopper.go:448 +0x4f

Previous write at 0x000011f03878 by goroutine 5320:
reflect.typedmemmove()
GOROOT/src/runtime/mbarrier.go:225 +0x0
reflect.Value.Set()
GOROOT/src/reflect/value.go:2321 +0x184
github.com/cockroachdb/cockroach/pkg/testutils.TestingHook.func1()
pkg/testutils/hook.go:18 +0x70
runtime.deferreturn()
GOROOT/src/runtime/panic.go:611 +0x5d
testing.tRunner()
GOROOT/src/testing/testing.go:1690 +0x226
testing.(*T).Run.gowrap1()
GOROOT/src/testing/testing.go:1743 +0x44

Goroutine 5323 (running) created at:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
pkg/util/stop/stopper.go:445 +0x24e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
pkg/util/stop/stopper.go:368 +0x1ef
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer.(*Balancer).processQueue.func1()
pkg/ccl/sqlproxyccl/balancer/balancer.go:366 +0xc0
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer.(*Balancer).processQueue()
pkg/ccl/sqlproxyccl/balancer/balancer.go:389 +0xa5
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer.(*Balancer).processQueue-fm()
:1 +0x47
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1()
pkg/util/stop/stopper.go:447 +0x115
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.gowrap1()
pkg/util/stop/stopper.go:448 +0x4f

Goroutine 5320 (finished) created at:
testing.(*T).Run()
GOROOT/src/testing/testing.go:1743 +0x825
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.TestCancelQuery()
pkg/ccl/sqlproxyccl/proxy_handler_test.go:1729 +0x1cfe
testing.tRunner()
GOROOT/src/testing/testing.go:1690 +0x226
testing.(*T).Run.gowrap1()
GOROOT/src/testing/testing.go:1743 +0x44
==================
```

Parameters:
- attempt=1
- race=true
- run=2
- shard=7
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #174774 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-cloud-platform branch-release-25.4.17-rc release-blocker]
- #172049 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-cloud-platform branch-release-26.1.6-rc]
- #169965 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-cloud-platform branch-release-25.4]
- #169046 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-cloud-platform branch-release-25.2.18-rc]
- #167577 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-cloud-platform branch-release-24.3]
- #166665 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-cloud-platform branch-release-24.3.30-rc]
- #165202 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-cloud-platform branch-release-26.1]
- #165007 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-cloud-platform branch-release-24.3.29-rc]
- #161165 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-cloud-platform branch-release-24.3.25-rc]

/cc @cockroachlabs/sqlproxy-prs

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestCancelQuery.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Start with pkg/ccl/sqlproxyccl/proxy_handler_test.go:1729 and reproduce ccl/sqlproxyccl.TestCancelQuery with race=true. Read the reported paths in conn_migration.go, balancer/balancer.go, and pkg/testutils/hook.go to trace the concurrent access. Done means the test no longer reports this race failure.

Written by the indexing model from the issue text.