cockroachdb / cockroachdb/cockroach

kv/kvnemesis: TestKVNemesisSingleNode failed

Open
#174,663 3 comments 0 reactions 0 assignees View on GitHub
A-kv A-kv-transactions branch-release-25.2 C-bug C-test-failure O-robot P-3 T-kv X-duplicate X-nostale
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

kv/kvnemesis.TestKVNemesisSingleNode [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsAwsLinuxArm64_UnitTests/21498289?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsAwsLinuxArm64_UnitTests/21498289?buildTab=artifacts#/) on release-25.2 @ [5cc7374f71df028e849db805e4e42e7cd6137bbc](https://github.com/cockroachdb/cockroach/commits/5cc7374f71df028e849db805e4e42e7cd6137bbc):

```
=== RUN TestKVNemesisSingleNode
test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/logTestKVNemesisSingleNode2546832374
test_log_scope.go:76: use -show-logs to present logs inline
kvnemesis_test.go:405: seed: 8738468950792482999
kvnemesis_test.go:246: kvnemesis logging to /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/kvnemesis980408325
kvnemesis.go:173: range 76 (/Table/100/"9616fc37f179f8de") RANGE_CONSISTENT_STATS_INCORRECT:
stats: {ContainsEstimates:0 LastUpdateNanos:1788446121280459737 LockAge:2 GCBytesAge:275 LiveBytes:200 LiveCount:2 KeyBytes:144 KeyCount:4 ValBytes:270 ValCount:5 IntentBytes:52 IntentCount:2 LockBytes:0 LockCount:2 RangeKeyCount:1 RangeKeyBytes:51 RangeValCount:1 RangeValBytes:10 SysBytes:688 SysCount:8 AbortSpanBytes:174}
delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1788446121280459737 LockAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 LockBytes:0 LockCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:20 SysCount:0 AbortSpanBytes:0}
kvnemesis.go:208: failures(verbose): /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/kvnemesis980408325/failures
repro steps: /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/kvnemesis980408325/repro.go
rangefeed KVs: /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/kvnemesis980408325/kvs-rangefeed.txt
scan KVs: /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/kvnemesis980408325/kvs-scan.txt
kvnemesis_test.go:438: [0] proposed: 987
kvnemesis_test.go:440: [0] reproposed unchanged: 74
kvnemesis_test.go:442: [0] reproposed with new LAI: 3
kvnemesis_test.go:447:
Error Trace: pkg/kv/kvnemesis/kvnemesis_test.go:447
pkg/kv/kvnemesis/kvnemesis_test.go:305
Error: Should be zero, but was 1
Test: TestKVNemesisSingleNode
Messages: kvnemesis detected failures
panic.go:635: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/logTestKVNemesisSingleNode2546832374
--- FAIL: TestKVNemesisSingleNode (14.15s)
```
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestKVNemesisSingleNode.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-67860

Contributor guide

Open the contributing guide

Research direction

Start with pkg/kv/kvnemesis/kvnemesis_test.go around the failure at lines 305 and 447, then inspect kvnemesis.go around the consistency check at lines 173 and 208. Use the linked build artifacts, especially the repro.go and failures files, to reproduce and investigate the single-node failure. Done means TestKVNemesisSingleNode completes without kvnemesis-detected failures or an incorrect range-stats delta.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.