basalt-26.3: roachtest: kv95/enc=false/nodes=3/cpu=32/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=enabled failed

roachtest.kv95/enc=false/nodes=3/cpu=32/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=enabled [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyBenchmarksAwsBasaltBazel/21516296?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyBenchmarksAwsBasaltBazel/21516296?buildTab=artifacts#/kv95/enc=false/nodes=3/cpu=32/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=enabled) on basalt-26.3 @ [2327de0124eca58e8ea82a577a805766789a48e5](https://github.com/cockroachdb/cockroach/commits/2327de0124eca58e8ea82a577a805766789a48e5):

Failed with:

```
(cluster.go:4230).Start: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=32/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=enabled/run_1
```
Fatal entries found in Cockroach logs:

```
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !goroutine 144 [running]:
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x1b0248965798?, 0xd6afcb?})
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:19
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x1b02489a2280}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x1b023e7700f0, 0x1, ...})
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/channels.go:104 +0x650
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !goroutine 144 [running]:
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x1b0248965798?, 0xd6afcb?})
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:19
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x1b02489a2280}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x1b023e7700f0, 0x1, ...})
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/channels.go:104 +0x650
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !goroutine 144 [running]:
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x1b0248965798?, 0xd6afcb?})
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:19
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x1b02489a2280}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x1b023e7700f0, 0x1, ...})
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/channels.go:104 +0x650
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 [n?,s?,pebble] 27 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !goroutine 144 [running]:
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x1b0248965798?, 0xd6afcb?})
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/allstacks/allstacks.go:19
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x1b02489a2280}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x1b023e7700f0, 0x1, ...})
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/log/channels.go:104 +0x650
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !goroutine 144 [running]:
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x1b0248965798?, 0xd6afcb?})
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:19
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x1b02489a2280}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x1b023e7700f0, 0x1, ...})
F260914 22:46:40.622695 144 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/channels.go:104 +0x650
```
Cluster Node to Ip Mapping:

| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21516296-1789423542-12-n4cpu32-0001 | 3.144.28.224 | 10.12.15.236 |
| teamcity-21516296-1789423542-12-n4cpu32-0002 | 3.147.84.107 | 10.12.2.155 |
| teamcity-21516296-1789423542-12-n4cpu32-0003 | 18.221.0.66 | 10.12.3.213 |
| teamcity-21516296-1789423542-12-n4cpu32-0004 | 3.135.187.244 | 10.12.10.213 |

Parameters:
- MetamorphicBasaltMultiAZ=false
- MetamorphicFilesystem=ext4
- arch=amd64
- cloud=aws
- coverageBuild=false
- cpu=32
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=false
- runtimeAssertionsBuild=false
- storageNodesVersion=v0.1.19-nightly.20260914
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21516296/kv95-enc-false-nodes-3-cpu-32-binary-basalt-storage-basalt-basa/1789425988902/1789426142506)

_Logs were not uploaded to Datadog_

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*kv95/enc=false/nodes=3/cpu=32/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=enabled.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68404

Start with the kv95 artifacts and logs linked in the report, then read pkg/cmd/roachtest/README.md and inspect the failure at pkg/storage/pebble/version_set.go:456. Determine why the basalt storage run reports an unsupported protocol version, reproduce the roachtest if possible, and confirm the test completes without fatal log entries.

Written by the indexing model from the issue text.