release-26.3.2-rc: crosscluster/physical: TestIngestSpanConfigsFullScan failed

crosscluster/physical.TestIngestSpanConfigsFullScan [failed](https://mesolite.cluster.engflow.com/invocations/default/de349425-8afc-41ea-af42-5f0a1623931e?testReportRun=9&testReportShard=14&testReportAttempt=1#targets-Ly9wa2cvY3Jvc3NjbHVzdGVyL3BoeXNpY2FsOnBoeXNpY2FsX3Rlc3Q=) on release-26.3.2-rc @ [8db696c79524f16c5fd046165e52a2e8c7eff326](https://github.com/cockroachdb/cockroach/commits/8db696c79524f16c5fd046165e52a2e8c7eff326):

```
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchStreamImpl(0x18f89f131808, {0x7f69e712ec60, 0x18f8b77b89b0})
pkg/server/node.go:1966 +0x90
github.com/cockroachdb/cockroach/pkg/server.(*Node).BatchStream(0x18f89f131808?, {0x806ff98?, 0x18f8b77b89b0?})
pkg/server/node.go:1953 +0x3e
github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_BatchStream_Handler({0x7a26580?, 0x18f89f131808}, {0x8057e90, 0x18f8bb202780})
bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:12315 +0xd8
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x7a26580, 0x18f89f131808}, {0x8057e90, 0x18f8bb202780}, 0x18f8c2585968, 0x7f6b1b0)
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:107 +0x50e
google.golang.org/grpc.getChainStreamHandler.func1({0x7a26580?, 0x18f89f131808?}, {0x8057e90?, 0x18f8bb202780?})
external/org_golang_google_grpc/server.go:1504 +0x153
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x7a26580, 0x18f89f131808}, {0x8057e90, 0x18f8bb202780}, 0x18f8c2585968?, 0x18f8bee974c0)
pkg/rpc/context.go:191 +0x63
google.golang.org/grpc.getChainStreamHandler.func1.getChainStreamHandler.1({0x7a26580, 0x18f89f131808}, {0x8057e90, 0x18f8bb202780})
external/org_golang_google_grpc/server.go:1504 +0xa8
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0x18f89eb10000, {{0x1}, {0x8080620, 0x18f89f064780}}, 0x0}, {0x7a26580, 0x18f89f131808}, {0x80583e8, 0x18f8bee851d0}, 0x18f8c2585968, ...)
pkg/rpc/auth.go:192 +0x3ba
google.golang.org/grpc.getChainStreamHandler.func1({0x7a26580?, 0x18f89f131808?}, {0x80583e8?, 0x18f8bee851d0?})
external/org_golang_google_grpc/server.go:1504 +0x153
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x77afd00?, 0x8015870?})
pkg/rpc/context.go:148 +0x27
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0x18f89e64f4d0, {0x8015870, 0x18f8bb2026f0}, {0x18f8b8c4a150, 0x27}, 0x18f8bf20baa8)
pkg/util/stop/stopper.go:358 +0x138
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x7a26580, 0x18f89f131808}, {0x80583e8, 0x18f8bee851d0}, 0x18f8c2585968, 0x18f8bee97440)
pkg/rpc/context.go:147 +0xcc
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x7a26580, 0x18f89f131808}, {0x80583e8, 0x18f8bee851d0}, 0x18f8c2585968, 0x18f8b77b8990?)
external/org_golang_google_grpc/server.go:1495 +0x7c
google.golang.org/grpc.(*Server).processStreamingRPC(0x18f89e868960, {0x8081820, 0x18f8b7a97ba0}, 0x18f8b7a56d80, 0x18f89e9577d0, 0xdd9f6a0, 0x0)
external/org_golang_google_grpc/server.go:1659 +0x1251
google.golang.org/grpc.(*Server).handleStream(0x18f89e868960, {0x8081820, 0x18f8b7a97ba0}, 0x18f8b7a56d80, 0x0)
external/org_golang_google_grpc/server.go:1739 +0x987
google.golang.org/grpc.(*Server).serveStreams.func1.1()
external/org_golang_google_grpc/server.go:970 +0xaf
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 3401
external/org_golang_google_grpc/server.go:981 +0x133

goroutine 46734 [select, "job":"AUTO SPAN CONFIG RECONCILIATION id=1210950798969241601", "n":"sql1"]:
github.com/cockroachdb/cockroach/pkg/rpc.(*pooledStream[...]).runOnce(0x807f420, {0x8015838, 0x18f8bee83900})
pkg/rpc/stream_pool.go:107 +0xe5
github.com/cockroachdb/cockroach/pkg/rpc.(*pooledStream[...]).run(0x807f420, {0x8015838, 0x18f8bee83900})
pkg/rpc/stream_pool.go:102 +0x78
github.com/cockroachdb/cockroach/pkg/rpc.(*streamPool[...]).newPooledStream.func2()
pkg/rpc/stream_pool.go:314 +0xa5
created by github.com/cockroachdb/cockroach/pkg/rpc.(*streamPool[...]).newPooledStream in goroutine 46731
pkg/rpc/stream_pool.go:312 +0x336

goroutine 41321 [select]:
github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
external/com_github_cockroachdb_pebble/vfs/disk_health.go:262 +0xb5
created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 543
external/com_github_cockroachdb_pebble/vfs/disk_health.go:257 +0x58
```

Parameters:
- attempt=1
- run=9
- shard=14
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #175119 crosscluster/physical: TestIngestSpanConfigsFullScan failed [C-test-failure O-robot branch-staging-v25.4.16 release-blocker]

/cc @cockroachlabs/disaster-recovery

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestIngestSpanConfigsFullScan.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68465

Start by rerunning crosscluster/physical.TestIngestSpanConfigsFullScan with attempt=1, run=9, and shard=14, then inspect the failure report and the stack frames in pkg/server/node.go and pkg/rpc/stream_pool.go. Compare the same failure in #175119 and the roachdash history; done means identifying and fixing the cause so the test passes on the affected release branch.

Written by the indexing model from the issue text.