basalt-26.3: roachtest: kv0/enc=false/nodes=3/size=64kb/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=enabled failed

roachtest.kv0/enc=false/nodes=3/size=64kb/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=enabled [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyBenchmarksAwsBasaltBazel/21516296?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyBenchmarksAwsBasaltBazel/21516296?buildTab=artifacts#/kv0/enc=false/nodes=3/size=64kb/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=enabled) on basalt-26.3 @ [2327de0124eca58e8ea82a577a805766789a48e5](https://github.com/cockroachdb/cockroach/commits/2327de0124eca58e8ea82a577a805766789a48e5):

Failed with:

```
(cluster.go:4230).Start: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/size=64kb/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=enabled/run_1
```
Fatal entries found in Cockroach logs:

```
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !goroutine 103 [running]:
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x28ed69c47798?, 0xd6afcb?})
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:19
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x28ed69bfe1e0}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x28ed67835c10, 0x1, ...})
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/channels.go:104 +0x650
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !goroutine 103 [running]:
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x28ed69c47798?, 0xd6afcb?})
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:19
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x28ed69bfe1e0}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x28ed67835c10, 0x1, ...})
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/channels.go:104 +0x650
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !goroutine 103 [running]:
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x28ed69c47798?, 0xd6afcb?})
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:19
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x28ed69bfe1e0}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x28ed67835c10, 0x1, ...})
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/channels.go:104 +0x650
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 [n?,s?,pebble] 27 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !goroutine 103 [running]:
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x28ed69c47798?, 0xd6afcb?})
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/allstacks/allstacks.go:19
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x28ed69bfe1e0}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x28ed67835c10, 0x1, ...})
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/log/channels.go:104 +0x650
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !goroutine 103 [running]:
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x28ed69c47798?, 0xd6afcb?})
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:19
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x28ed69bfe1e0}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x28ed67835c10, 0x1, ...})
F260914 22:27:04.153538 103 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/channels.go:104 +0x650
```
Cluster Node to Ip Mapping:

| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21516296-1789423542-06-n4cpu8-0001 | 18.222.216.246 | 10.12.17.19 |
| teamcity-21516296-1789423542-06-n4cpu8-0002 | 3.143.141.54 | 10.12.20.119 |
| teamcity-21516296-1789423542-06-n4cpu8-0003 | 18.116.117.152 | 10.12.28.84 |
| teamcity-21516296-1789423542-06-n4cpu8-0004 | 3.140.250.139 | 10.12.18.124 |

Parameters:
- MetamorphicBasaltMultiAZ=false
- MetamorphicFilesystem=ext4
- arch=amd64
- cloud=aws
- coverageBuild=false
- cpu=8
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=false
- runtimeAssertionsBuild=false
- storageNodesVersion=v0.1.19-nightly.20260914
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21516296/kv0-enc-false-nodes-3-size-64kb-binary-basalt-storage-basalt-ba/1789424811929/1789424966373)

_Logs were not uploaded to Datadog_

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*kv0/enc=false/nodes=3/size=64kb/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=enabled.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68398

Start with the failed roachtest artifacts and logs at /artifacts/kv0/enc=false/nodes=3/size=64kb/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=enabled/run_1, then read pkg/cmd/roachtest/README.md. Investigate the MANIFEST flush failure reported at pebble/version_set.go:456 and use the linked TeamCity run and Grafana report for context. Done means the basalt kv0 configuration completes without the unsupported protocol version fatal error.

Written by the indexing model from the issue text.