basalt-26.3: roachtest: tpcc/headroom/n4cpu16/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=enabled failed

roachtest.tpcc/headroom/n4cpu16/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=enabled [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyBenchmarksAwsBasaltBazel/21516296?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyBenchmarksAwsBasaltBazel/21516296?buildTab=artifacts#/tpcc/headroom/n4cpu16/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=enabled) on basalt-26.3 @ [2327de0124eca58e8ea82a577a805766789a48e5](https://github.com/cockroachdb/cockroach/commits/2327de0124eca58e8ea82a577a805766789a48e5):

Failed with:

```
(cluster.go:4230).Start: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/tpcc/headroom/n4cpu16/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=enabled/run_1
```
Fatal entries found in Cockroach logs:

```
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 MANIFEST flush failed: version 0: unsupported protocol version
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !goroutine 102 [running]:
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0xf0e5b38f798?, 0xd6afcb?})
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:19
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/clog.go:294 +0xc6
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0xf0e5b42a500}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0xf0e513ec2e0, 0x1, ...})
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/channels.go:104 +0x650
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 MANIFEST flush failed: version 0: unsupported protocol version
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !goroutine 102 [running]:
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0xf0e5b38f798?, 0xd6afcb?})
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:19
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/clog.go:294 +0xc6
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0xf0e5b42a500}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0xf0e513ec2e0, 0x1, ...})
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/channels.go:104 +0x650
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 MANIFEST flush failed: version 0: unsupported protocol version
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !goroutine 102 [running]:
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0xf0e5b38f798?, 0xd6afcb?})
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:19
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/clog.go:294 +0xc6
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0xf0e5b42a500}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0xf0e513ec2e0, 0x1, ...})
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/channels.go:104 +0x650
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 [n?,s?,pebble] 27 MANIFEST flush failed: version 0: unsupported protocol version
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !goroutine 102 [running]:
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0xf0e5b38f798?, 0xd6afcb?})
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/allstacks/allstacks.go:19
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/log/clog.go:294 +0xc6
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0xf0e5b42a500}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0xf0e513ec2e0, 0x1, ...})
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/log/channels.go:104 +0x650
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 MANIFEST flush failed: version 0: unsupported protocol version
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !goroutine 102 [running]:
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0xf0e5b38f798?, 0xd6afcb?})
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:19
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/clog.go:294 +0xc6
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0xf0e5b42a500}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0xf0e513ec2e0, 0x1, ...})
F260914 23:34:12.119546 102 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/channels.go:104 +0x650
```
Cluster Node to Ip Mapping:

| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21516296-1789423542-25-n4cpu16-0001 | 18.191.85.114 | 10.12.2.32 |
| teamcity-21516296-1789423542-25-n4cpu16-0002 | 18.223.203.250 | 10.12.15.223 |
| teamcity-21516296-1789423542-25-n4cpu16-0003 | 16.58.213.217 | 10.12.14.187 |
| teamcity-21516296-1789423542-25-n4cpu16-0004 | 3.145.152.245 | 10.12.14.70 |

Parameters:
- MetamorphicBasaltMultiAZ=false
- MetamorphicFilesystem=ext4
- arch=amd64
- cloud=aws
- coverageBuild=false
- cpu=16
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=false
- metamorphicLeases=default
- runtimeAssertionsBuild=false
- storageNodesVersion=v0.1.19-nightly.20260914
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21516296/tpcc-headroom-n4cpu16-binary-basalt-storage-basalt-basalt-nodes/1789428830932/1789428996332)

_Logs were not uploaded to Datadog_

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*tpcc/headroom/n4cpu16/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=enabled.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68413

Start with the roachtest README and the failure at cluster.go:4230, then inspect the linked TeamCity artifacts and logs for the tpcc/headroom basalt run. Investigate the Pebble MANIFEST flush failure reporting an unsupported protocol version. Done means the roachtest no longer fails with this error and the relevant test run passes.

Written by the indexing model from the issue text.