cockroachdb / cockroachdb/cockroach
basalt-26.3.2-rc-20260914: kv/kvnemesis: TestKVNemesisMultiNode_Crash failed
Open
Nobody has claimed this yet.
branch-basalt-26.3.2-rc-20260914
C-test-failure
O-robot
release-blocker
T-kv
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
kv/kvnemesis.TestKVNemesisMultiNode_Crash failed on basalt-26.3.2-rc-20260914 @ d3eb08ebdd1dacc8d3dac0b107a7db5929e09969:
=== RUN TestKVNemesisMultiNode_Crash
test_log_scope.go:172: test logs captured to: outputs.zip/logTestKVNemesisMultiNode_Crash2487505870
test_log_scope.go:82: use -show-logs to present logs inline
kvnemesis_test.go:797: seed: 1297684151769582746
kvnemesis_test.go:332: n1 using clock[offset=-16.286µs,amp=3.483714ms]
kvnemesis_test.go:332: n2 using clock[offset=-23.851µs,amp=3.476149ms]
kvnemesis_test.go:332: n3 using clock[offset=52.647µs,amp=3.447353ms]
kvnemesis_test.go:332: n4 using clock[offset=-160.244µs,amp=3.339756ms]
test_server_shim.go:146: using test tenant configuration from testserver factory defaults
kvnemesis_test.go:411: kvnemesis logging to /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/testlogs/pkg/kv/kvnemesis/kvnemesis_test/run_9_of_10/test.outputs/kvnemesis2278997136
kvnemesis.go:241: error applying x.AdminSplit(ctx, tk(17940889579152782081), hlc.MaxTimestamp) // split at key /Table/100/"f8fad8046fe8b301" failed: replica unavailable: (n1,s1):1 unable to serve request to r2:/{Meta2-System/NodeLiveness} [(n1,s1):1, (n3,s3):4, (n4,s4):3, next=5, gen=8]: lost quorum (down: (n3,s3):4,(n4,s4):3); closed timestamp: 1789881906.965922814,0 (2026-09-20 05:25:06); raft status: {"id":"1","term":8,"vote":"1","commit":1185,"lead":"0","leadEpoch":"0","raftState":"StatePreCandidate","applied":1185,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Meta2/""]: split at key /Table/100/"f8fad8046fe8b301" failed: replica unavailable: (n1,s1):1 unable to serve request to r2:/{Meta2-System/NodeLiveness} [(n1,s1):1, (n3,s3):4, (n4,s4):3, next=5, gen=8]: lost quorum (down: (n3,s3):4,(n4,s4):3); closed timestamp: 1789881906.965922814,0 (2026-09-20 05:25:06); raft status: {"id":"1","term":8,"vote":"1","commit":1185,"lead":"0","leadEpoch":"0","raftState":"StatePreCandidate","applied":1185,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Meta2/""]
kvnemesis.go:282: failures(verbose): /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/testlogs/pkg/kv/kvnemesis/kvnemesis_test/run_9_of_10/test.outputs/kvnemesis2278997136/failures
repro steps: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/testlogs/pkg/kv/kvnemesis/kvnemesis_test/run_9_of_10/test.outputs/kvnemesis2278997136/repro.go
rangefeed KVs: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/testlogs/pkg/kv/kvnemesis/kvnemesis_test/run_9_of_10/test.outputs/kvnemesis2278997136/kvs-rangefeed.txt
scan KVs: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/testlogs/pkg/kv/kvnemesis/kvnemesis_test/run_9_of_10/test.outputs/kvnemesis2278997136/kvs-scan.txt
kvnemesis_test.go:927: Metric | Node 1 | Node 2 | Node 3 | Node 4
kvnemesis_test.go:933: ------------------------------------+--------------------------------+--------------------------------+--------------------------------+-------------------------------
kvnemesis_test.go:940: follower_reads.success_count | 1 | 109 | 1 | 0
kvnemesis_test.go:940: raft.commands.proposed | 10500 | 230 | 4 | 5
kvnemesis_test.go:940: raft.commands.reproposed.new-lai | 7 | 0 | 0 | 0
kvnemesis_test.go:940: raft.commands.reproposed.unchanged | 204 | 18 | 2 | 4
kvnemesis_test.go:940: txn.aborts | 23 | 39 | 1 | 1
kvnemesis_test.go:940: txn.commits | 1783 | 903 | 108 | 90
kvnemesis_test.go:940: txn.durations | μ=28ms p50=5ms p99=168ms | μ=48ms p50=10ms p99=419ms | μ=23ms p50=5ms p99=50ms | μ=27ms p50=6ms p99=55ms
kvnemesis_test.go:940: txn.restarts.writetooold | 5 | 1 | 0 | 0
kvnemesis_test.go:940: txn.restarts.serializable | 1 | 0 | 0 | 0
kvnemesis_test.go:940: txn.restarts.readwithinuncertainty | 0 | 0 | 0 | 0
kvnemesis_test.go:940: txn.server_side.1PC.success | 172 | 19 | 0 | 0
kvnemesis_test.go:940: txnrecovery.failures | 0 | 0 | 0 | 0
kvnemesis_test.go:940: txnrecovery.successes.aborted | 0 | 0 | 0 | 0
kvnemesis_test.go:940: txnrecovery.successes.committed | 0 | 0 | 0 | 0
kvnemesis_test.go:940: txnwaitqueue.deadlocks_total | 31 | 0 | 0 | 0
kvnemesis_test.go:860:
Error Trace: pkg/kv/kvnemesis/kvnemesis_test.go:860
pkg/kv/kvnemesis/kvnemesis_test.go:658
Error: Should be zero, but was 1
Test: TestKVNemesisMultiNode_Crash
Messages: kvnemesis detected failures
dump_raft_logs.go:32: dumping raft logs to /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/testlogs/pkg/kv/kvnemesis/kvnemesis_test/run_9_of_10/test.outputs/kvnemesis2278997136/raftlogs
panic.go:694: -- test log scope end --
test logs left over in: outputs.zip/logTestKVNemesisMultiNode_Crash2487505870
--- FAIL: TestKVNemesisMultiNode_Crash (116.89s)
Parameters:
attempt=1run=9shard=1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with pkg/kv/kvnemesis/kvnemesis_test.go and TestKVNemesisMultiNode_Crash, using seed 1297684151769582746 and the linked failure report. Inspect the generated repro.go, failures, and raft logs to determine why the multi-node crash test loses quorum. Done means the cause is addressed and the test passes reliably.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100