upgrade/upgrades: TestFirstUpgradeRepairBatchSize failed

upgrade/upgrades.TestFirstUpgradeRepairBatchSize [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsAwsLinuxArm64_UnitTests/21404814?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsAwsLinuxArm64_UnitTests/21404814?buildTab=artifacts#/) on master @ [7bb28d6af41a04f1dfbf1780b2c35b4a33735607](https://github.com/cockroachdb/cockroach/commits/7bb28d6af41a04f1dfbf1780b2c35b4a33735607):

```
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2.1({0x5bd3090?, 0x5cfa515f5dc0?})
pkg/kv/kvserver/intentresolver/intent_resolver.go:937 +0x38 fp=0x5cfa469d7e20 sp=0x5cfa469d7de0 pc=0x3e80d18
github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout({0x5bd2cc0?, 0xa792220?}, {0x578d502, 0x12}, 0x4a817c800, 0x5cfa4e7c7f80)
pkg/util/timeutil/timeout.go:28 +0x80 fp=0x5cfa469d7eb0 sp=0x5cfa469d7e20 pc=0x973720
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2({0x5bd2cc0, 0xa792220})
pkg/kv/kvserver/intentresolver/intent_resolver.go:935 +0xc4 fp=0x5cfa469d7fb0 sp=0x5cfa469d7eb0 pc=0x3e80c14
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.gowrap1()
pkg/kv/kvserver/intentresolver/intent_resolver.go:947 +0x28 fp=0x5cfa469d7fd0 sp=0x5cfa469d7fb0 pc=0x3e80b38
runtime.goexit({})
src/runtime/asm_arm64.s:1447 +0x4 fp=0x5cfa469d7fd0 sp=0x5cfa469d7fd0 pc=0x4a33d4
created by github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents in goroutine 81998
pkg/kv/kvserver/intentresolver/intent_resolver.go:933 +0x228

goroutine 59160 gp=0x5cfa586fef00 m=nil [select, "tags":"n1,rnode=1,raddr=127.0.0.1:39527,class=system,rpc", "workload.id":"RPC_HEARTBEAT"]:
runtime.gopark(0x5cfa48535ef8?, 0x2?, 0x78?, 0x5d?, 0x5cfa48535ec4?)
GOROOT/src/runtime/proc.go:677 +0xbc fp=0x5cfa48535d40 sp=0x5cfa48535d20 pc=0x49abfc
runtime.selectgo(0x5cfa48535ef8, 0x5cfa48535ec0, 0x5cfa48535ec8?, 0x0, 0x5cfa54a48c90?, 0x1)
GOROOT/src/runtime/select.go:351 +0x83c fp=0x5cfa48535e80 sp=0x5cfa48535d40 pc=0x47805c
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x5cfa49777a40, 0x1)
external/org_golang_google_grpc/internal/transport/controlbuf.go:418 +0x144 fp=0x5cfa48535f20 sp=0x5cfa48535e80 pc=0xb35e24
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x5cfa58e98620)
external/org_golang_google_grpc/internal/transport/controlbuf.go:552 +0x6c fp=0x5cfa48535f80 sp=0x5cfa48535f20 pc=0xb3660c
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
external/org_golang_google_grpc/internal/transport/http2_client.go:451 +0x64 fp=0x5cfa48535fd0 sp=0x5cfa48535f80 pc=0xb3e624
runtime.goexit({})
src/runtime/asm_arm64.s:1447 +0x4 fp=0x5cfa48535fd0 sp=0x5cfa48535fd0 pc=0x4a33d4
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 59204
external/org_golang_google_grpc/internal/transport/http2_client.go:449 +0x17b4

goroutine 81826 gp=0x5cfa5a0a0d20 m=nil [select, "job":"AUTO CREATE STATS id=1188416684851331073", "n":"1"]:
runtime.gopark(0x5cfa4be27c28?, 0x3?, 0x50?, 0xef?, 0x5cfa4be27bf2?)
GOROOT/src/runtime/proc.go:677 +0xbc fp=0x5cfa47e73a40 sp=0x5cfa47e73a20 pc=0x49abfc
runtime.selectgo(0x5cfa47e73c28, 0x5cfa4be27bec, 0x5cfa535d65b0?, 0x0, 0x5cfa4849fc08?, 0x1)
GOROOT/src/runtime/select.go:351 +0x83c fp=0x5cfa47e73b80 sp=0x5cfa47e73a40 pc=0x47805c
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0x5cfa33535200, {0x5bd3090, 0x5cfa535d65b0}, 0x50, {0x5c3c618, 0x5cfa331b9b00})
pkg/internal/client/requestbatcher/batcher.go:339 +0x108 fp=0x5cfa47e73c80 sp=0x5cfa47e73b80 pc=0x3e7a0b8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).gcTxnRecord(0x5cfa332f6c60, {0x5bd3090, 0x5cfa535d65b0}, 0x50, 0x5cfa4afd9320)
pkg/kv/kvserver/intentresolver/intent_resolver.go:888 +0x2a0 fp=0x5cfa47e73de0 sp=0x5cfa47e73c80 pc=0x3e80740
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2.1({0x5bd3090?, 0x5cfa535d65b0?})
pkg/kv/kvserver/intentresolver/intent_resolver.go:937 +0x38 fp=0x5cfa47e73e20 sp=0x5cfa47e73de0 pc=0x3e80d18
github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout({0x5bd2cc0?, 0xa792220?}, {0x578d502, 0x12}, 0x4a817c800, 0x5cfa4be27f80)
pkg/util/timeutil/timeout.go:28 +0x80 fp=0x5cfa47e73eb0 sp=0x5cfa47e73e20 pc=0x973720
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2({0x5bd2cc0, 0xa792220})
pkg/kv/kvserver/intentresolver/intent_resolver.go:935 +0xc4 fp=0x5cfa47e73fb0 sp=0x5cfa47e73eb0 pc=0x3e80c14
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.gowrap1()
pkg/kv/kvserver/intentresolver/intent_resolver.go:947 +0x28 fp=0x5cfa47e73fd0 sp=0x5cfa47e73fb0 pc=0x3e80b38
runtime.goexit({})
src/runtime/asm_arm64.s:1447 +0x4 fp=0x5cfa47e73fd0 sp=0x5cfa47e73fd0 pc=0x4a33d4
created by github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents in goroutine 81728
pkg/kv/kvserver/intentresolver/intent_resolver.go:933 +0x228
```
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachlabs/release-eng @cockroachlabs/release-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestFirstUpgradeRepairBatchSize.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-65257

Start by reviewing the failing upgrade/upgrades.TestFirstUpgradeRepairBatchSize run and its TeamCity log and artifacts. Then inspect pkg/kv/kvserver/intentresolver/intent_resolver.go around lines 888 and 933-947, where the stack trace shows transaction cleanup and timeout handling. Done means the failure is understood and the test passes reliably on the affected run.

Written by the indexing model from the issue text.