cockroachdb / cockroachdb/cockroach

Sentry: syncing_file.go:65: write ×: cannot allocate memory (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:770 | github.com/cockroachdb/cockroach/pk...

Open
#172,470 1 comment 0 reactions 0 assignees View on GitHub
A-storage branch-release-24.1 C-bug O-sentry T-storage X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7605611514/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7605611514/?referrer=webhooks_plugin)

Panic Message:

```
syncing_file.go:65: write ×: cannot allocate memory
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:770
| github.com/cockroachdb/cockroach/pkg/storage.(*writeBatch).SyncWait
| github.com/cockroachdb/cockroach/pkg/storage/pebble_batch.go:384
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*SyncWaiterLoop).waitLoop
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/sync_waiter.go:95
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*SyncWaiterLoop).Start.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/sync_waiter.go:84
| [...repeated from below...]
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/pebble/vfs.(*syncingFile).Write
| github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/syncing_file.go:65
| github.com/cockroachdb/pebble/record.(*LogWriter).flushBlock
| github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:782
| github.com/cockroachdb/pebble/record.(*LogWriter).flushPending
| github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:749
| github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop
| github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:695
| runtime/pprof.Do
| GOROOT/src/runtime/pprof/runtime.go:51
| github.com/cockroachdb/pebble/record.NewLogWriter.func2
| github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:562
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (3) write ×
Wraps: (4) cannot allocate memory
Error types: (1) *withstack.withStack (2) *withstack.withStack (3) *fs.PathError (4) syscall.Errno
-- report composition:
syscall.Errno
*fs.PathError
syncing_file.go:65: *withstack.withStack (top exception)
panic.go:770: *withstack.withStack (1)
(check the extra data payloads)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[https://github.com/cockroachdb/cockroach/blob/5e4ca9e26f1a25681de9c944298cfa139c344466/pkg/util/stop/stopper.go#L484-L486](https://github.com/cockroachdb/cockroach/blob/5e4ca9e26f1a25681de9c944298cfa139c344466/pkg/util/stop/stopper.go#L484-L486)
[https://github.com/cockroachdb/cockroach/blob/5e4ca9e26f1a25681de9c944298cfa139c344466/pkg/kv/kvserver/logstore/sync_waiter.go#L83-L85](https://github.com/cockroachdb/cockroach/blob/5e4ca9e26f1a25681de9c944298cfa139c344466/pkg/kv/kvserver/logstore/sync_waiter.go#L83-L85)
[https://github.com/cockroachdb/cockroach/blob/5e4ca9e26f1a25681de9c944298cfa139c344466/pkg/kv/kvserver/logstore/sync_waiter.go#L94-L96](https://github.com/cockroachdb/cockroach/blob/5e4ca9e26f1a25681de9c944298cfa139c344466/pkg/kv/kvserver/logstore/sync_waiter.go#L94-L96)
[https://github.com/cockroachdb/cockroach/blob/5e4ca9e26f1a25681de9c944298cfa139c344466/pkg/storage/pebble_batch.go#L383-L385](https://github.com/cockroachdb/cockroach/blob/5e4ca9e26f1a25681de9c944298cfa139c344466/pkg/storage/pebble_batch.go#L383-L385)
[GOROOT/src/runtime/panic.go#L769-L771](GOROOT/src/runtime/panic.go#L769-L771)
[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L561-L563](github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L561-L563)
[GOROOT/src/runtime/pprof/runtime.go#L50-L52](GOROOT/src/runtime/pprof/runtime.go#L50-L52)
[github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L694-L696](github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L694-L696)
[github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L748-L750](github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L748-L750)
[github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L781-L783](github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L781-L783)
[github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/syncing_file.go#L64-L66](github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/syncing_file.go#L64-L66)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 485
pkg/kv/kvserver/logstore/sync_waiter.go in pkg/kv/kvserver/logstore.(*SyncWaiterLoop).Start.func1 at line 84
pkg/kv/kvserver/logstore/sync_waiter.go in pkg/kv/kvserver/logstore.(*SyncWaiterLoop).waitLoop at line 95
pkg/storage/pebble_batch.go in pkg/storage.(*writeBatch).SyncWait at line 384
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
src/runtime/asm_amd64.s in runtime.goexit at line 1695
github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.NewLogWriter.func2 at line 562
GOROOT/src/runtime/pprof/runtime.go in runtime/pprof.Do at line 51
github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop at line 695
github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushPending at line 749
github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushBlock at line 782
github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/syncing_file.go in github.com/cockroachdb/pebble/vfs.(*syncingFile).Write at line 65
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v24.1.0 |
| Go Version | go1.22.2 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v24.1.0 |
| Cockroach SHA | 5e4ca9e26f1a25681de9c944298cfa139c344466 |
| # of CPUs | 24 |
| # of Goroutines | 632 |

Jira issue: CRDB-65708

Contributor guide

Open the contributing guide

Research direction

Start with pkg/storage/pebble_batch.go and pkg/kv/kvserver/logstore/sync_waiter.go, then follow the reported write path into the Pebble syncing_file.go and log_writer.go references. Determine why the allocation failure reaches SyncWait as a panic and identify a reproducible test or diagnostic path; done requires a confirmed cause and an agreed fix scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.