cockroachdb / cockroachdb/pebble

Intermittent panic during a `Get` call while processing a lot of `Set` batch commits

Open
#5,002 3 comments 0 reactions 0 assignees View on GitHub
A-storage O-community T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

Long time listener, first time caller.
Not sure if this is related to the v2 upgrade or what but this started happening for me when running some workloads with heavy Get's and batch Set's at the same time. I'm not sure if y'all know if it's likely a me problem (I have all my iterators and closers wrapped up by defers) or what might be causing this? I never got this on the v1 pebble but that might be coincidence. I only have been able to reproduce a couple of time which leads me to believe it's some sort of concurrency problem.

```
Recovered from panic: runtime error: invalid memory address or nil pointer dereference
Stack trace:
goroutine 268 [running]:
runtime/debug.Stack()
[...]
/opt/homebrew/Cellar/go/1.24.4/libexec/src/runtime/panic.go:792 +0x124
github.com/cockroachdb/pebble/v2/sstable.(*twoLevelIterator[...]).loadSecondLevelIndexBlock(0x102a84b80, 0x1)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.6/sstable/reader_iter_two_lvl.go:76 +0x25c
github.com/cockroachdb/pebble/v2/sstable.(*twoLevelIterator[...]).skipForward(0x102a84b80)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.6/sstable/reader_iter_two_lvl.go:946 +0xd8
github.com/cockroachdb/pebble/v2/sstable.(*twoLevelIterator[...]).SeekPrefixGE(0x102a84b80, {0x1400c1fa708, 0x12, 0x12}, {0x1400c1fa708, 0x140006c2fb8?, 0x101389558?}, 0x0?)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.6/sstable/reader_iter_two_lvl.go:587 +0x448
github.com/cockroachdb/pebble/v2.(*getIter).Next(0x14000a815f0)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.6/get_iter.go:133 +0x214
github.com/cockroachdb/pebble/v2.(*getIter).First(0x140006c3058?)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.6/get_iter.go:71 +0x1c
github.com/cockroachdb/pebble/v2.(*Iterator).iterFirstWithinBounds(0x14000a81008)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.6/iterator.go:2099 +0x90
github.com/cockroachdb/pebble/v2.(*Iterator).First(0x14000a81008)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.6/iterator.go:1651 +0xe4
github.com/cockroachdb/pebble/v2.(*DB).getInternal(0x14000dc6908, {0x1400c1fa708, 0x12, 0x12}, 0x0, 0x0)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.6/db.go:627 +0x330
github.com/cockroachdb/pebble/v2.(*DB).Get(...)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.6/db.go:545
[....]
golang.org/x/sync/errgroup.(*Group).add.func1()
/Users/ajroetker/go/pkg/mod/golang.org/x/sync@v0.15.0/errgroup/errgroup.go:128 +0x88
created by golang.org/x/sync/errgroup.(*Group).add in goroutine 227
/Users/ajroetker/go/pkg/mod/golang.org/x/sync@v0.15.0/errgroup/errgroup.go:96 +0x80
```

Jira issue: PEBBLE-525

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.