cockroachdb / cockroachdb/cockroach
batcheval: investigate latch assertion in DeleteRange
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
With timestamp-aware latch assertions enabled in #158131, stressing `kvnemesis` catches the following error. Need to investigate and fix. Reproduces easily with the linked PR: `dev test --stress pkg/kv/kvnemesis`.
A couple of quirks to be aware:
- `/{Min-Table/100/"05ee0fb26ba7b41b"}` is a `[x.Prev(), x)` span. Does not actually start at `Min`, it's just a pretty-printer quirk (need to fix).
- The `???` key is probably something like a "prevish" [key](https://github.com/cockroachdb/cockroach/blob/1f1ee11a76efe5f7be61f82836131d605ebed155/pkg/kv/kvserver/batcheval/ranges.go#L17) which the pretty-printer doesn't understand well.
Previously, the assertion did not fire either because we weren't stressing `kvnemesis` under `--race`, or because it was more liberal (did not take timestamps into account).
What happens here is: in the `read global`, the `???` is probably the prevish key of `"05ee0fb26ba7b41b"`. It is an MVCC span. Previously, we ignored its timestamp, now we don't. So maybe it doesn't like the non-MVCC `SeekLT`. Rules are [here](https://github.com/cockroachdb/cockroach/blob/ef4e9e488e7889a4193d814150dc27646438c0b6/pkg/kv/kvserver/spanset/spanset.go#L300-L324).
```
kvnemesis.go:240: error applying x.DelRangeUsingTombstone(ctx, tk(427296273631523867), tk(15658150965789664794) /* @s71 */) // cannot read undeclared span /{Min-Table/100/"05ee0fb26ba7b41b"}
declared:
read global: /Table/100/{???-"d94ced57cb55261a"/NULL} at 1763636207.304303935,0
read local: /Local/RangeID/79/u"rltu" at 0,0
read local: /Local/Range/Table/76/RangeDescriptor at 0,0
read local: /Local/Lock/Table/100/{???-"d94ced57cb55261a"/NULL} at 0,0
read local: /Local/Lock/Local/RangeID/79/u"rltu" at 0,0
read local: /Local/Lock/Local/Range/Table/76/RangeDescriptor at 0,0
write global: /Table/100/"{05ee0fb26ba7b41b"-d94ced57cb55261a"} at 1763636207.304303935,0
write local: /Local/Lock/Table/100/"{05ee0fb26ba7b41b"-d94ced57cb55261a"} at 0,0
stack:
goroutine 187735 [running]:
runtime/debug.Stack()
GOROOT/src/runtime/debug/stack.go:26 +0x64
github.com/cockroachdb/cockroach/pkg/util/debugutil.Stack(...)
pkg/util/debugutil/debugutil.go:63
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*SpanSet).checkAllowed(0x14003cd88c0, 0x0, {{0x0, 0x0, 0x0}, {0x1401274a780, 0x14, 0x18}}, 0x14012a4a628)
pkg/kv/kvserver/spanset/spanset.go:408 +0x134
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*SpanSet).CheckAllowedAt(0x401?, 0x401?, {{0x0, 0x0, 0x0}, {0x1401274a780, 0x14, 0x18}}, {0x14012a4a6e8?, 0x2112f4c?})
pkg/kv/kvserver/spanset/spanset.go:294 +0x90
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*MVCCIterator).checkAllowedValidPos(0x1401446d360, {{0x0, 0x0, 0x0}, {0x1401274a780, 0x14, 0x18}}, 0x1)
pkg/kv/kvserver/spanset/batch.go:142 +0xa4
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*MVCCIterator).checkAllowed(0x1401446d360, {{0x0, 0x0, 0x0}, {0x1401274a780, 0x14, 0x18}}, 0x1)
pkg/kv/kvserver/spanset/batch.go:134 +0x78
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*MVCCIterator).SeekLT(0x1401446d360, {{0x1401274a780, 0x14, 0x18}, {0x0, 0x0}})
pkg/kv/kvserver/spanset/batch.go:88 +0x6c
github.com/cockroachdb/cockroach/pkg/storage.PeekRangeKeysLeft({0x1066ac2c0, 0x1401446d360}, {0x1401274a780, 0x14, 0x18})
pkg/storage/mvcc.go:8336 +0x68
github.com/cockroachdb/cockroach/pkg/storage.MVCCDeleteRangeUsingTombstone({0x1066276f8, 0x140135c8f30}, {0x1066bf950, 0x1400d332fc0}, 0x1401339e630, {0x1401274a780, 0x14, 0x18}, {0x1401274a798, 0x14, ...}, ...)
pkg/storage/mvcc.go:4316 +0x1270
github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.deleteRangeUsingTombstone({_, _}, {_, _}, {{0x1066c69e8, 0x1400ae10580}, {{0x1879b1c9a9a47011, 0x3}, 0x140127ec700, {0x1879b1c94de988c6, ...}, ...}, ...})
pkg/kv/kvserver/batcheval/cmd_delete_range.go:181 +0x2c0
github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.DeleteRange({_, _}, {_, _}, {{0x1066c69e8, 0x1400ae10580}, {{0x1879b1c9a9a47011, 0x3}, 0x140127ec700, {0x1879b1c94de988c6, ...}, ...}, ...}, ...)
pkg/kv/kvserver/batcheval/cmd_delete_range.go:123 +0x33c
```
Jira issue: CRDB-57124
Contributor guide
Research direction
Reproduce the failure with `dev test --stress pkg/kv/kvnemesis`, then trace `DeleteRangeUsingTombstone` in pkg/storage/mvcc.go through cmd_delete_range.go and the span checks in pkg/kv/kvserver/spanset/spanset.go and batch.go. Compare the timestamp-aware rules with the non-MVCC SeekLT path and use the linked stress case to validate the fix. Done means the assertion no longer fires under the stress test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100