cockroachdb / cockroachdb/cockroach
basalt-26.3: roachtest: kv0/enc=false/nodes=3/size=64kb/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=disabled failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.kv0/enc=false/nodes=3/size=64kb/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=disabled [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyBenchmarksAwsBasaltBazel/21516296?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyBenchmarksAwsBasaltBazel/21516296?buildTab=artifacts#/kv0/enc=false/nodes=3/size=64kb/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=disabled) on basalt-26.3 @ [2327de0124eca58e8ea82a577a805766789a48e5](https://github.com/cockroachdb/cockroach/commits/2327de0124eca58e8ea82a577a805766789a48e5):
Failed with:
```
(cluster.go:4230).Start: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/size=64kb/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=disabled/run_1
```
Fatal entries found in Cockroach logs:
```
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !goroutine 96 [running]:
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x3c785e90b798?, 0xd6afcb?})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:19
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x3c785ea06460}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x3c785c5699f0, 0x1, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/channels.go:104 +0x650
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !goroutine 96 [running]:
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x3c785e90b798?, 0xd6afcb?})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/allstacks/allstacks.go:19
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x3c785ea06460}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x3c785c5699f0, 0x1, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 8 ! pkg/util/log/channels.go:104 +0x650
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !goroutine 96 [running]:
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x3c785e90b798?, 0xd6afcb?})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:19
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x3c785ea06460}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x3c785c5699f0, 0x1, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/channels.go:104 +0x650
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !goroutine 96 [running]:
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x3c785e90b798?, 0xd6afcb?})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/allstacks/allstacks.go:19
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x3c785ea06460}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x3c785c5699f0, 0x1, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 [n?,s?,pebble] 27 ! pkg/util/log/channels.go:104 +0x650
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 MANIFEST flush failed: version 0: unsupported protocol version
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !goroutine 96 [running]:
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x3c785e90b798?, 0xd6afcb?})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/allstacks/allstacks.go:19
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/clog.go:294 +0xc6
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86e51c8, 0x3c785ea06460}, 0x3, 0x4, 0x3, 0x0?, {0x8630d28, 0x2}, {0x3c785c5699f0, 0x1, ...})
F260914 22:22:47.434175 96 3@pebble/version_set.go:456 ⋮ [n?,s?,pebble] 25 ! pkg/util/log/channels.go:104 +0x650
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21516296-1789423542-05-n4cpu8-0001 | 3.21.240.140 | 10.12.21.79 |
| teamcity-21516296-1789423542-05-n4cpu8-0002 | 18.191.83.196 | 10.12.23.59 |
| teamcity-21516296-1789423542-05-n4cpu8-0003 | 3.140.192.255 | 10.12.24.9 |
| teamcity-21516296-1789423542-05-n4cpu8-0004 | 18.222.136.48 | 10.12.22.151 |
Parameters:
- MetamorphicBasaltMultiAZ=false
- MetamorphicFilesystem=xfs
- arch=amd64
- cloud=aws
- coverageBuild=false
- cpu=8
- diskCount=0
- encrypted=false
- fs=xfs
- localSSD=false
- runtimeAssertionsBuild=false
- storageNodesVersion=v0.1.19-nightly.20260914
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21516296/kv0-enc-false-nodes-3-size-64kb-binary-basalt-storage-basalt-ba/1789424555643/1789424709108)
_Logs were not uploaded to Datadog_
/cc @cockroachlabs/test-eng
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*kv0/enc=false/nodes=3/size=64kb/binary=basalt/storage=basalt/basalt_nodes=3/rsengine=disabled.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-68397
Contributor guide
Research direction
Start with the roachtest artifacts and logs linked in the report, then inspect pkg/cmd/roachtest/README.md and the reported locations cluster.go:4230 and pebble/version_set.go:456. Determine why the basalt storage test encounters an unsupported protocol version and verify that the kv0 configuration completes without fatal log entries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud, databases, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100