cockroachdb / cockroachdb/cockroach
Sentry: syncing_file.go:65: write ×: Insufficient system resources exist to complete the requested service. (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/pani...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/7341081012/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7341081012/?referrer=webhooks_plugin)
Panic Message:
```
syncing_file.go:65: write ×: Insufficient system resources exist to complete the requested service.
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:770
| github.com/cockroachdb/pebble.(*DB).commitWrite
| external/com_github_cockroachdb_pebble/db.go:987
| github.com/cockroachdb/pebble.(*commitPipeline).prepare
| external/com_github_cockroachdb_pebble/commit.go:468
| github.com/cockroachdb/pebble.(*commitPipeline).Commit
| external/com_github_cockroachdb_pebble/commit.go:317
| github.com/cockroachdb/pebble.(*DB).applyInternal
| external/com_github_cockroachdb_pebble/db.go:872
| github.com/cockroachdb/pebble.(*DB).ApplyNoSyncWait
| external/com_github_cockroachdb_pebble/db.go:820
| github.com/cockroachdb/cockroach/pkg/storage.(*writeBatch).CommitNoSyncWait
| pkg/storage/pebble_batch.go:353
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch
| pkg/kv/kvserver/logstore/logstore.go:307
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).StoreEntries
| pkg/kv/kvserver/logstore/logstore.go:203
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1235
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| pkg/kv/kvserver/replica_raft.go:846
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:682
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:419
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:319
| [...repeated from below...]
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/pebble/vfs.(*syncingFile).Write
| external/com_github_cockroachdb_pebble/vfs/syncing_file.go:65
| github.com/cockroachdb/pebble/record.(*LogWriter).flushBlock
| external/com_github_cockroachdb_pebble/record/log_writer.go:783
| github.com/cockroachdb/pebble/record.(*LogWriter).flushPending
| external/com_github_cockroachdb_pebble/record/log_writer.go:750
| github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop
| external/com_github_cockroachdb_pebble/record/log_writer.go:696
| runtime/pprof.Do
| GOROOT/src/runtime/pprof/runtime.go:51
| github.com/cockroachdb/pebble/record.NewLogWriter.func2
| external/com_github_cockroachdb_pebble/record/log_writer.go:563
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (3) write ×
Wraps: (4) Insufficient system resources exist to complete the requested service.
Error types: (1) *withstack.withStack (2) *withstack.withStack (3) *fs.PathError (4) syscall.Errno
-- report composition:
syscall.Errno
*fs.PathError
syncing_file.go:65: *withstack.withStack (top exception)
panic.go:770: *withstack.withStack (1)
(check the extra data payloads)
```
Stacktrace (expand for inline code snippets):
[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[pkg/util/stop/stopper.go#L497-L499](pkg/util/stop/stopper.go#L497-L499)
[pkg/kv/kvserver/scheduler.go#L318-L320](pkg/kv/kvserver/scheduler.go#L318-L320)
[pkg/kv/kvserver/scheduler.go#L418-L420](pkg/kv/kvserver/scheduler.go#L418-L420)
[pkg/kv/kvserver/store_raft.go#L681-L683](pkg/kv/kvserver/store_raft.go#L681-L683)
[pkg/kv/kvserver/replica_raft.go#L845-L847](pkg/kv/kvserver/replica_raft.go#L845-L847)
[pkg/kv/kvserver/replica_raft.go#L1234-L1236](pkg/kv/kvserver/replica_raft.go#L1234-L1236)
[pkg/kv/kvserver/logstore/logstore.go#L202-L204](pkg/kv/kvserver/logstore/logstore.go#L202-L204)
[pkg/kv/kvserver/logstore/logstore.go#L306-L308](pkg/kv/kvserver/logstore/logstore.go#L306-L308)
[pkg/storage/pebble_batch.go#L352-L354](pkg/storage/pebble_batch.go#L352-L354)
[external/com_github_cockroachdb_pebble/db.go#L819-L821](external/com_github_cockroachdb_pebble/db.go#L819-L821)
[external/com_github_cockroachdb_pebble/db.go#L871-L873](external/com_github_cockroachdb_pebble/db.go#L871-L873)
[external/com_github_cockroachdb_pebble/commit.go#L316-L318](external/com_github_cockroachdb_pebble/commit.go#L316-L318)
[external/com_github_cockroachdb_pebble/commit.go#L467-L469](external/com_github_cockroachdb_pebble/commit.go#L467-L469)
[external/com_github_cockroachdb_pebble/db.go#L986-L988](external/com_github_cockroachdb_pebble/db.go#L986-L988)
[GOROOT/src/runtime/panic.go#L769-L771](GOROOT/src/runtime/panic.go#L769-L771)
[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[external/com_github_cockroachdb_pebble/record/log_writer.go#L562-L564](external/com_github_cockroachdb_pebble/record/log_writer.go#L562-L564)
[GOROOT/src/runtime/pprof/runtime.go#L50-L52](GOROOT/src/runtime/pprof/runtime.go#L50-L52)
[external/com_github_cockroachdb_pebble/record/log_writer.go#L695-L697](external/com_github_cockroachdb_pebble/record/log_writer.go#L695-L697)
[external/com_github_cockroachdb_pebble/record/log_writer.go#L749-L751](external/com_github_cockroachdb_pebble/record/log_writer.go#L749-L751)
[external/com_github_cockroachdb_pebble/record/log_writer.go#L782-L784](external/com_github_cockroachdb_pebble/record/log_writer.go#L782-L784)
[external/com_github_cockroachdb_pebble/vfs/syncing_file.go#L64-L66](external/com_github_cockroachdb_pebble/vfs/syncing_file.go#L64-L66)
```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 498
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 319
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 419
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 682
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 846
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1235
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).StoreEntries at line 203
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch at line 307
pkg/storage/pebble_batch.go in pkg/storage.(*writeBatch).CommitNoSyncWait at line 353
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).ApplyNoSyncWait at line 820
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).applyInternal at line 872
external/com_github_cockroachdb_pebble/commit.go in github.com/cockroachdb/pebble.(*commitPipeline).Commit at line 317
external/com_github_cockroachdb_pebble/commit.go in github.com/cockroachdb/pebble.(*commitPipeline).prepare at line 468
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).commitWrite at line 987
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
src/runtime/asm_amd64.s in runtime.goexit at line 1695
external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.NewLogWriter.func2 at line 563
GOROOT/src/runtime/pprof/runtime.go in runtime/pprof.Do at line 51
external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop at line 696
external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushPending at line 750
external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushBlock at line 783
external/com_github_cockroachdb_pebble/vfs/syncing_file.go in github.com/cockroachdb/pebble/vfs.(*syncingFile).Write at line 65
```
### Tags
| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v24.3.1 |
| Go Version | go1.22.8 X:nocoverageredesign |
| Platform | windows amd64 |
| Distribution | CCL |
| Cockroach Release | v24.3.1 |
| Cockroach SHA | 950be288db97fe1d8f9c19918e37695d3c1f4bb0 |
| # of CPUs | 8 |
| # of Goroutines | 514 |
Jira issue: CRDB-61695
Contributor guide
Assessment
This issue has not been assessed yet.