cockroachdb / cockroachdb/cockroach

basalt-26.3: roachtest: kv0/enc=false/nodes=3/size=64kb/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=disabled failed

Closed
#175,479 1 comment 0 reactions 0 assignees View on GitHub
branch-basalt-26.3 C-test-failure O-roachtest O-robot release-blocker T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.kv0/enc=false/nodes=3/size=64kb/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=disabled [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyBenchmarksAwsBasaltBazel/21516296?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyBenchmarksAwsBasaltBazel/21516296?buildTab=artifacts#/kv0/enc=false/nodes=3/size=64kb/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=disabled) on basalt-26.3 @ [2327de0124eca58e8ea82a577a805766789a48e5](https://github.com/cockroachdb/cockroach/commits/2327de0124eca58e8ea82a577a805766789a48e5):

Failed with:

```
(cluster.go:4230).Start: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/size=64kb/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=disabled/run_1
```
Fatal entries found in Cockroach logs:

```
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !goroutine 96 [running]:
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x3c785e90b798?, 0xd6afcb?})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:19
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x3c785ea06460}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x3c785c5699f0, 0x1, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/channels.go:104 +0x650
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !goroutine 96 [running]:
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x3c785e90b798?, 0xd6afcb?})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:19
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x3c785ea06460}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x3c785c5699f0, 0x1, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/channels.go:104 +0x650
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !goroutine 96 [running]:
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x3c785e90b798?, 0xd6afcb?})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:19
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x3c785ea06460}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x3c785c5699f0, 0x1, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/channels.go:104 +0x650
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !goroutine 96 [running]:
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x3c785e90b798?, 0xd6afcb?})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/allstacks/allstacks.go:19
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x3c785ea06460}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x3c785c5699f0, 0x1, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/log/channels.go:104 +0x650
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !goroutine 96 [running]:
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x3c785e90b798?, 0xd6afcb?})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:19
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x3c785ea06460}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x3c785c5699f0, 0x1, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/channels.go:104 +0x650
```
Cluster Node to Ip Mapping:

| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21516296-1789423542-05-n4cpu8-0001 | 3.21.240.140 | 10.12.21.79 |
| teamcity-21516296-1789423542-05-n4cpu8-0002 | 18.191.83.196 | 10.12.23.59 |
| teamcity-21516296-1789423542-05-n4cpu8-0003 | 3.140.192.255 | 10.12.24.9 |
| teamcity-21516296-1789423542-05-n4cpu8-0004 | 18.222.136.48 | 10.12.22.151 |

Parameters:
- MetamorphicBasaltMultiAZ=false
- MetamorphicFilesystem=xfs
- arch=amd64
- cloud=aws
- coverageBuild=false
- cpu=8
- diskCount=0
- encrypted=false
- fs=xfs
- localSSD=false
- runtimeAssertionsBuild=false
- storageNodesVersion=v0.1.19-nightly.20260914
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21516296/kv0-enc-false-nodes-3-size-64kb-binary-basalt-storage-basalt-ba/1789424555643/1789424709108)

_Logs were not uploaded to Datadog_

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*kv0/enc=false/nodes=3/size=64kb/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=disabled.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68397

Contributor guide

Open the contributing guide

Research direction

Start with the roachtest artifacts and logs linked in the report, then inspect pkg/cmd/roachtest/README.md and the reported locations cluster.go:4230 and pebble/version_set.go:456. Determine why the basalt storage test encounters an unsupported protocol version and verify that the kv0 configuration completes without fatal log entries.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud, databases, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.