basalt-26.3: roachtest: import/nodeShutdown/worker/distmerge=false/nodes=4 failed

roachtest.import/nodeShutdown/worker/distmerge=false/nodes=4 [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_GreenRoachtestNightlyGcePrivateBasaltBazel/21514658?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_GreenRoachtestNightlyGcePrivateBasaltBazel/21514658?buildTab=artifacts#/import/nodeShutdown/worker/distmerge=false/nodes=4) on basalt-26.3 @ [4fda4910b2015319b2f775e42d2f2fb15130d19a](https://github.com/cockroachdb/cockroach/commits/4fda4910b2015319b2f775e42d2f2fb15130d19a):

Failed with:

```
(test_runner.go:2762).func1: task ``task-2`` returned error: import tpch/customer: could not restart node :3: failed to wait for SQL to be ready: dial tcp 10.91.100.67:26257: connect: connection refused
test artifacts and logs in: /artifacts/import/nodeShutdown/worker/distmerge=false/nodes=4/run_1
```
Fatal entries found in Cockroach logs:

```
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 while ingesting ‹auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested›: create ‹auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested›: grpc: ‹file "i34.t6.ingested" already exists› [code 6/AlreadyExists]
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !goroutine 1150 [running]:
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x36f77ad9f018?, 0xd6afcb?})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 ! pkg/util/allstacks/allstacks.go:19
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x36f77d770cc0, 0x24}, {0x82309ac, 0x1}, {0x82309aa, 0x1}, {0x8137de9, 0x6}, {0x82309ac, ...}}, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 ! pkg/util/log/clog.go:294 +0xc6
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86df4c8, 0x36f7821bc960}, 0x2, 0x4, 0xf, 0x0?, {0x8124688, 0x2}, {0x36f77ad9f898, 0x1, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 ! pkg/util/log/channels.go:104 +0x650
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 while ingesting ‹auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested›: create ‹auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested›: grpc: ‹file "i34.t6.ingested" already exists› [code 6/AlreadyExists]
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !goroutine 1150 [running]:
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x36f77ad9f018?, 0xd6afcb?})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 ! pkg/util/allstacks/allstacks.go:19
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x36f77d770cc0, 0x24}, {0x82309ac, 0x1}, {0x82309aa, 0x1}, {0x8137de9, 0x6}, {0x82309ac, ...}}, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 ! pkg/util/log/clog.go:294 +0xc6
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86df4c8, 0x36f7821bc960}, 0x2, 0x4, 0xf, 0x0?, {0x8124688, 0x2}, {0x36f77ad9f898, 0x1, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 ! pkg/util/log/channels.go:104 +0x650
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 while ingesting ‹auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested›: create ‹auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested›: grpc: ‹file "i34.t6.ingested" already exists› [code 6/AlreadyExists]
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !goroutine 1150 [running]:
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x36f77ad9f018?, 0xd6afcb?})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 ! pkg/util/allstacks/allstacks.go:19
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x36f77d770cc0, 0x24}, {0x82309ac, 0x1}, {0x82309aa, 0x1}, {0x8137de9, 0x6}, {0x82309ac, ...}}, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 ! pkg/util/log/clog.go:294 +0xc6
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86df4c8, 0x36f7821bc960}, 0x2, 0x4, 0xf, 0x0?, {0x8124688, 0x2}, {0x36f77ad9f898, 0x1, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 ! pkg/util/log/channels.go:104 +0x650
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 while ingesting auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested: create auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested: grpc: file "i34.t6.ingested" already exists [code 6/AlreadyExists]
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 !goroutine 1150 [running]:
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x36f77ad9f018?, 0xd6afcb?})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 ! pkg/util/allstacks/allstacks.go:19
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x36f77d770cc0, 0x24}, {0x82309ac, 0x1}, {0x82309aa, 0x1}, {0x8137de9, 0x6}, {0x82309ac, ...}}, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 ! pkg/util/log/clog.go:294 +0xc6
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86df4c8, 0x36f7821bc960}, 0x2, 0x4, 0xf, 0x0?, {0x8124688, 0x2}, {0x36f77ad9f898, 0x1, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 ! pkg/util/log/channels.go:104 +0x650
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 while ingesting ‹auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested›: create ‹auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested›: grpc: ‹file "i34.t6.ingested" already exists› [code 6/AlreadyExists]
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !goroutine 1150 [running]:
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x36f77ad9f018?, 0xd6afcb?})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 ! pkg/util/allstacks/allstacks.go:19
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x36f77d770cc0, 0x24}, {0x82309ac, 0x1}, {0x82309aa, 0x1}, {0x8137de9, 0x6}, {0x82309ac, ...}}, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 ! pkg/util/log/clog.go:294 +0xc6
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86df4c8, 0x36f7821bc960}, 0x2, 0x4, 0xf, 0x0?, {0x8124688, 0x2}, {0x36f77ad9f898, 0x1, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 ! pkg/util/log/channels.go:104 +0x650
```
Cluster Node to Ip Mapping:

| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21514658-1789279845-98-n4cpu4-0001 | | 10.91.100.68 |
| teamcity-21514658-1789279845-98-n4cpu4-0002 | | 10.91.99.214 |
| teamcity-21514658-1789279845-98-n4cpu4-0003 | | 10.91.100.67 |
| teamcity-21514658-1789279845-98-n4cpu4-0004 | | 10.91.99.211 |

Parameters:
- MetamorphicBasaltMultiAZ=false
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=true
- fs=ext4
- localSSD=true
- metamorphicLeases=default
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=false
- storageNodesVersion=v0.1.19-nightly.20260912
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21514658/import-nodeshutdown-worker-distmerge-false-nodes-4/1789288902171/1789289533617)

_Logs were not uploaded to Datadog_

/cc @cockroachlabs/sql-foundations

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*import/nodeShutdown/worker/distmerge=false/nodes=4.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Start with the roachtest.import/nodeShutdown/worker/distmerge=false/nodes=4 artifacts and logs, then inspect pkg/kv/kvserver/replica_proposal.go around line 763. Reproduce the basalt failure and trace the duplicate sideloading file error through the node restart path. Done means the roachtest no longer fails with the fatal AlreadyExists error and the restarted node becomes SQL-ready.

Written by the indexing model from the issue text.