cockroachdb / cockroachdb/pebble

fatal during high volume read/writes: `fatal error: unexpected signal during runtime execution`

Open
#5,245 2 comments 0 reactions 0 assignees View on GitHub
A-storage O-community T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

Not sure if this is user error or something in pebble but it only started happening after upgrading yesterday, although I've been improving my code's performance so it wouldn't be surprising if the difference was higher volume read writing tipping pebble over. Any insights what might be happening?

```
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x2 addr=0xf84307fef85f8405 pc=0x104fea47c]
goroutine 1452212
gp=0x1400129afc0 m=29 mp=0x140052ec808 [running]:
runtime.throw({0x106704508?, 0x0?})
/opt/homebrew/Cellar/go/1.25.0/libexec/src/runtime/panic.go:1094 +0x34 fp=0x14010a93cc0 sp=0x14010a93c90 pc=0x10502c304
runtime.sigpanic()
/opt/homebrew/Cellar/go/1.25.0/libexec/src/runtime/signal_unix.go:909 +0x350 fp=0x14010a93d20 sp=0x14010a93cc0 pc=0x10502e980
runtime.fpTracebackPartialExpand(0x2?, 0x41d?, {0x14005a84d80, 0x87, 0x14010a93e28?})
/opt/homebrew/Cellar/go/1.25.0/libexec/src/runtime/mprof.go:592 +0x3c fp=0x14010a93da0 sp=0x14010a93d30 pc=0x104fea47c
runtime.saveblockevent(0xcb2, 0x1, 0x4, 0x2)
/opt/homebrew/Cellar/go/1.25.0/libexec/src/runtime/mprof.go:563 +0x180 fp=0x14010a93df0 sp=0x14010a93da0 pc=0x104fea350
runtime.blockevent(0x1087a8520?, 0x3)
/opt/homebrew/Cellar/go/1.25.0/libexec/src/runtime/mprof.go:513 +0x5c fp=0x14010a93e30 sp=0x14010a93df0 pc=0x10502ae6c
runtime.semacquire1(0x14000f3e444, 0x0, 0x3, 0x0, 0x17)
/opt/homebrew/Cellar/go/1.25.0/libexec/src/runtime/sema.go:198 +0x2e4 fp=0x14010a93e80 sp=0x14010a93e30 pc=0x10500b384
sync.runtime_SemacquireRWMutexR(0x1409cdb8dc0?, 0x0?, 0x1402d8dc820?)
/opt/homebrew/Cellar/go/1.25.0/libexec/src/runtime/sema.go:100 +0x28 fp=0x14010a93ec0 sp=0x14010a93e80 pc=0x10502de28
sync.(*RWMutex).RLock(...)
/opt/homebrew/Cellar/go/1.25.0/libexec/src/sync/rwmutex.go:74
github.com/cockroachdb/pebble/v2/internal/cache.(*shard).Get(0x14000f3e428, 0x2, 0x41d, 0x3085e)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.7/internal/cache/clockpro.go:120 +0x78 fp=0x14010a93f10 sp=0x14010a93ec0 pc=0x1054f1ee8
github.com/cockroachdb/pebble/v2/internal/cache.(*Cache).Get(0x0?, 0x2, 0x41d, 0x3085e)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.7/internal/cache/clockpro.go:825 +0x38 fp=0x14010a93f40 sp=0x14010a93f10 pc=0x1054f40d8
github.com/cockroachdb/pebble/v2/sstable.(*Reader).readBlockInternal(0x14014902008, {0x10722ddc0, 0x1087c4680}, {0x1409cdb8dc0, 0x140064c49d8, 0x0}, {0x10722fb68, 0x140064c4908}, {0x0?, 0x3085e?}, ...)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.7/sstable/reader.go:500 +0x74 fp=0x14010a94150 sp=0x14010a93f40 pc=0x105645b54
github.com/cockroachdb/pebble/v2/sstable.(*Reader).readDataBlock(0x11?, {0x10722ddc0?, 0x1087c4680?}, {0x1409cdb8dc0?, 0x140064c49d8?, 0x0?}, {0x10722fb68?, 0x140064c4908?}, {0x0?, 0x140132394c0?})
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.7/sstable/reader.go:407 +0x48 fp=0x14010a941d0 sp=0x14010a94150 pc=0x105645618
github.com/cockroachdb/pebble/v2/sstable.(*singleLevelIterator[...]).loadDataBlock(0x10726c220, 0x1)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.7/sstable/reader_iter_single_lvl.go:567 +0x24c fp=0x14010a942c0 sp=0x14010a941d0 pc=0x10565bd0c
github.com/cockroachdb/pebble/v2/sstable.(*singleLevelIterator[...]).seekGEHelper(0x10726c220, {0x140132394c0, 0x40, 0x40}, 0x0, 0x0)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.7/sstable/reader_iter_single_lvl.go:826 +0x294 fp=0x14010a94330 sp=0x14010a942c0 pc=0x10565b2c4
github.com/cockroachdb/pebble/v2/sstable.(*singleLevelIterator[...]).SeekGE(0x10726c220, {0x140132394c0, 0x1054d3c30?, 0x14017e5dbb8?}, 0x0?)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.7/sstable/reader_iter_single_lvl.go:752 +0x150 fp=0x14010a94390 sp=0x14010a94330 pc=0x10565b5f0
github.com/cockroachdb/pebble/v2/sstable.(*twoLevelIterator[...]).SeekGE(0x107269f40, {0x140132394c0, 0x0?, 0x1409cdb9890?}, 0x0?)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.7/sstable/reader_iter_two_lvl.go:417 +0x33c fp=0x14010a94430 sp=0x14010a94390 pc=0x105658aec
github.com/cockroachdb/pebble/v2/sstable.(*twoLevelIterator[...]).SeekGE({0x140132394c0?, 0x1?, 0x40?}, 0x1?)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.7/sstable/reader_iter_two_lvl.go:289 +0x40 fp=0x14010a94470 sp=0x14010a94430 pc=0x105665c40
github.com/cockroachdb/pebble/v2.(*levelIter).SeekGE(0x1409cdb9810, {0x140132394c0, 0x40, 0x40}, 0x0)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.7/level_iter.go:657 +0xc4 fp=0x14010a944b0 sp=0x14010a94470 pc=0x1056fd8a4
github.com/cockroachdb/pebble/v2.(*mergingIter).seekGE(0x1409cdb9038, {0x140132394c0?, 0x0?, 0x0?}, 0x0?, 0x0)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.7/merging_iter.go:978 +0x16c fp=0x14010a94570 sp=0x14010a944b0 pc=0x1057051fc
github.com/cockroachdb/pebble/v2.(*mergingIter).SeekGE(0x1409cdb9038, {0x140132394c0?, 0x0?, 0x1400393d740?}, 0x0?)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.7/merging_iter.go:1026 +0x4c fp=0x14010a945b0 sp=0x14010a94570 pc=0x1057054cc
github.com/cockroachdb/pebble/v2.(*Iterator).SeekGEWithLimit(0x1409cdb8a08, {0x140132394c0, 0x40, 0x40}, {0x0, 0x0, 0x0})
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.7/iterator.go:1348 +0x470 fp=0x14010a94660 sp=0x14010a945b0 pc=0x1056f28d0
github.com/cockroachdb/pebble/v2.(*Iterator).SeekGE(...)
/Users/ajroetker/go/pkg/mod/github.com/cockroachdb/pebble/v2@v2.0.7/iterator.go:1252
....
```

Jira issue: PEBBLE-1155

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the stack trace in internal/cache/clockpro.go, sstable/reader.go, and the iterator files named in the report, then compare behavior before and after the upgrade. Reproduce the failure under high-volume reads and writes if possible, and determine whether Pebble or the Go runtime is responsible. Done means a confirmed cause and a validated fix or clear diagnosis.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.