cockroachdb / cockroachdb/cockroach

Sentry: syncing_file.go:65: write ×: Insufficient system resources exist to complete the requested service. (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/pani...

Open
#172,860 1 comment 0 reactions 0 assignees View on GitHub
A-storage branch-release-24.3 C-bug O-sentry T-storage X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7636314434/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7636314434/?referrer=webhooks_plugin)

Panic Message:

```
syncing_file.go:65: write ×: Insufficient system resources exist to complete the requested service.
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:770
| github.com/cockroachdb/pebble.(*DB).commitWrite
| external/com_github_cockroachdb_pebble/db.go:987
| github.com/cockroachdb/pebble.(*commitPipeline).prepare
| external/com_github_cockroachdb_pebble/commit.go:468
| github.com/cockroachdb/pebble.(*commitPipeline).Commit
| external/com_github_cockroachdb_pebble/commit.go:317
| github.com/cockroachdb/pebble.(*DB).applyInternal
| external/com_github_cockroachdb_pebble/db.go:872
| github.com/cockroachdb/pebble.(*DB).ApplyNoSyncWait
| external/com_github_cockroachdb_pebble/db.go:820
| github.com/cockroachdb/cockroach/pkg/storage.(*writeBatch).CommitNoSyncWait
| pkg/storage/pebble_batch.go:353
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch
| pkg/kv/kvserver/logstore/logstore.go:307
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).StoreEntries
| pkg/kv/kvserver/logstore/logstore.go:203
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1235
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| pkg/kv/kvserver/replica_raft.go:846
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:682
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:419
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:319
| [...repeated from below...]
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/pebble/vfs.(*syncingFile).Write
| external/com_github_cockroachdb_pebble/vfs/syncing_file.go:65
| github.com/cockroachdb/pebble/record.(*LogWriter).flushBlock
| external/com_github_cockroachdb_pebble/record/log_writer.go:783
| github.com/cockroachdb/pebble/record.(*LogWriter).flushPending
| external/com_github_cockroachdb_pebble/record/log_writer.go:750
| github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop
| external/com_github_cockroachdb_pebble/record/log_writer.go:696
| runtime/pprof.Do
| GOROOT/src/runtime/pprof/runtime.go:51
| github.com/cockroachdb/pebble/record.NewLogWriter.func2
| external/com_github_cockroachdb_pebble/record/log_writer.go:563
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (3) write ×
Wraps: (4) Insufficient system resources exist to complete the requested service.
Error types: (1) *withstack.withStack (2) *withstack.withStack (3) *fs.PathError (4) syscall.Errno
-- report composition:
syscall.Errno
*fs.PathError
syncing_file.go:65: *withstack.withStack (top exception)
panic.go:770: *withstack.withStack (1)
(check the extra data payloads)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[pkg/util/stop/stopper.go#L497-L499](pkg/util/stop/stopper.go#L497-L499)
[pkg/kv/kvserver/scheduler.go#L318-L320](pkg/kv/kvserver/scheduler.go#L318-L320)
[pkg/kv/kvserver/scheduler.go#L418-L420](pkg/kv/kvserver/scheduler.go#L418-L420)
[pkg/kv/kvserver/store_raft.go#L681-L683](pkg/kv/kvserver/store_raft.go#L681-L683)
[pkg/kv/kvserver/replica_raft.go#L845-L847](pkg/kv/kvserver/replica_raft.go#L845-L847)
[pkg/kv/kvserver/replica_raft.go#L1234-L1236](pkg/kv/kvserver/replica_raft.go#L1234-L1236)
[pkg/kv/kvserver/logstore/logstore.go#L202-L204](pkg/kv/kvserver/logstore/logstore.go#L202-L204)
[pkg/kv/kvserver/logstore/logstore.go#L306-L308](pkg/kv/kvserver/logstore/logstore.go#L306-L308)
[pkg/storage/pebble_batch.go#L352-L354](pkg/storage/pebble_batch.go#L352-L354)
[external/com_github_cockroachdb_pebble/db.go#L819-L821](external/com_github_cockroachdb_pebble/db.go#L819-L821)
[external/com_github_cockroachdb_pebble/db.go#L871-L873](external/com_github_cockroachdb_pebble/db.go#L871-L873)
[external/com_github_cockroachdb_pebble/commit.go#L316-L318](external/com_github_cockroachdb_pebble/commit.go#L316-L318)
[external/com_github_cockroachdb_pebble/commit.go#L467-L469](external/com_github_cockroachdb_pebble/commit.go#L467-L469)
[external/com_github_cockroachdb_pebble/db.go#L986-L988](external/com_github_cockroachdb_pebble/db.go#L986-L988)
[GOROOT/src/runtime/panic.go#L769-L771](GOROOT/src/runtime/panic.go#L769-L771)
[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[external/com_github_cockroachdb_pebble/record/log_writer.go#L562-L564](external/com_github_cockroachdb_pebble/record/log_writer.go#L562-L564)
[GOROOT/src/runtime/pprof/runtime.go#L50-L52](GOROOT/src/runtime/pprof/runtime.go#L50-L52)
[external/com_github_cockroachdb_pebble/record/log_writer.go#L695-L697](external/com_github_cockroachdb_pebble/record/log_writer.go#L695-L697)
[external/com_github_cockroachdb_pebble/record/log_writer.go#L749-L751](external/com_github_cockroachdb_pebble/record/log_writer.go#L749-L751)
[external/com_github_cockroachdb_pebble/record/log_writer.go#L782-L784](external/com_github_cockroachdb_pebble/record/log_writer.go#L782-L784)
[external/com_github_cockroachdb_pebble/vfs/syncing_file.go#L64-L66](external/com_github_cockroachdb_pebble/vfs/syncing_file.go#L64-L66)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 498
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 319
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 419
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 682
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 846
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1235
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).StoreEntries at line 203
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch at line 307
pkg/storage/pebble_batch.go in pkg/storage.(*writeBatch).CommitNoSyncWait at line 353
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).ApplyNoSyncWait at line 820
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).applyInternal at line 872
external/com_github_cockroachdb_pebble/commit.go in github.com/cockroachdb/pebble.(*commitPipeline).Commit at line 317
external/com_github_cockroachdb_pebble/commit.go in github.com/cockroachdb/pebble.(*commitPipeline).prepare at line 468
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).commitWrite at line 987
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
src/runtime/asm_amd64.s in runtime.goexit at line 1695
external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.NewLogWriter.func2 at line 563
GOROOT/src/runtime/pprof/runtime.go in runtime/pprof.Do at line 51
external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop at line 696
external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushPending at line 750
external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushBlock at line 783
external/com_github_cockroachdb_pebble/vfs/syncing_file.go in github.com/cockroachdb/pebble/vfs.(*syncingFile).Write at line 65
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v24.3.1 |
| Go Version | go1.22.8 X:nocoverageredesign |
| Platform | windows amd64 |
| Distribution | CCL |
| Cockroach Release | v24.3.1 |
| Cockroach SHA | 950be288db97fe1d8f9c19918e37695d3c1f4bb0 |
| # of CPUs | 4 |
| # of Goroutines | 358 |

Jira issue: CRDB-66128

Contributor guide

Open the contributing guide

Research direction

Start with the Windows start-single-node environment and trace the reported path from pkg/kv/kvserver/logstore/logstore.go through pkg/storage/pebble_batch.go to syncing_file.go:65. Reproduce or characterize the insufficient-resources failure, then determine the appropriate fix and verify that the crash no longer occurs with focused coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.