cockroachdb / cockroachdb/cockroach
Sentry: syncing_file.go:65: write ×: There is not enough space on the disk. (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:791 | github.com/cockroac...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/7689462230/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7689462230/?referrer=webhooks_plugin)
Panic Message:
```
syncing_file.go:65: write ×: There is not enough space on the disk.
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| github.com/cockroachdb/pebble.(*DB).commitWrite
| external/com_github_cockroachdb_pebble/db.go:990
| github.com/cockroachdb/pebble.(*commitPipeline).prepare
| external/com_github_cockroachdb_pebble/commit.go:468
| github.com/cockroachdb/pebble.(*commitPipeline).Commit
| external/com_github_cockroachdb_pebble/commit.go:317
| github.com/cockroachdb/pebble.(*DB).applyInternal
| external/com_github_cockroachdb_pebble/db.go:875
| github.com/cockroachdb/pebble.(*DB).ApplyNoSyncWait
| external/com_github_cockroachdb_pebble/db.go:823
| github.com/cockroachdb/cockroach/pkg/storage.(*writeBatch).CommitNoSyncWait
| pkg/storage/pebble_batch.go:378
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch
| pkg/kv/kvserver/logstore/logstore.go:263
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).StoreEntries
| pkg/kv/kvserver/logstore/logstore.go:159
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaLogStorage).appendRaftMuLocked
| pkg/kv/kvserver/replica_raftlog_writes.go:61
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1251
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| pkg/kv/kvserver/replica_raft.go:875
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:755
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:419
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:319
| [...repeated from below...]
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/pebble/vfs.(*syncingFile).Write
| external/com_github_cockroachdb_pebble/vfs/syncing_file.go:65
| github.com/cockroachdb/pebble/record.(*LogWriter).flushBlock
| external/com_github_cockroachdb_pebble/record/log_writer.go:812
| github.com/cockroachdb/pebble/record.(*LogWriter).flushPending
| external/com_github_cockroachdb_pebble/record/log_writer.go:779
| github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop
| external/com_github_cockroachdb_pebble/record/log_writer.go:724
| runtime/pprof.Do
| GOROOT/src/runtime/pprof/runtime.go:51
| github.com/cockroachdb/pebble/record.NewLogWriter.func2
| external/com_github_cockroachdb_pebble/record/log_writer.go:585
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (3) write ×
Wraps: (4) There is not enough space on the disk.
Error types: (1) *withstack.withStack (2) *withstack.withStack (3) *fs.PathError (4) syscall.Errno
-- report composition:
syscall.Errno
*fs.PathError
syncing_file.go:65: *withstack.withStack (top exception)
panic.go:791: *withstack.withStack (1)
(check the extra data payloads)
```
Stacktrace (expand for inline code snippets):
[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[pkg/util/stop/stopper.go#L499-L501](pkg/util/stop/stopper.go#L499-L501)
[pkg/kv/kvserver/scheduler.go#L318-L320](pkg/kv/kvserver/scheduler.go#L318-L320)
[pkg/kv/kvserver/scheduler.go#L418-L420](pkg/kv/kvserver/scheduler.go#L418-L420)
[pkg/kv/kvserver/store_raft.go#L754-L756](pkg/kv/kvserver/store_raft.go#L754-L756)
[pkg/kv/kvserver/replica_raft.go#L874-L876](pkg/kv/kvserver/replica_raft.go#L874-L876)
[pkg/kv/kvserver/replica_raft.go#L1250-L1252](pkg/kv/kvserver/replica_raft.go#L1250-L1252)
[pkg/kv/kvserver/replica_raftlog_writes.go#L60-L62](pkg/kv/kvserver/replica_raftlog_writes.go#L60-L62)
[pkg/kv/kvserver/logstore/logstore.go#L158-L160](pkg/kv/kvserver/logstore/logstore.go#L158-L160)
[pkg/kv/kvserver/logstore/logstore.go#L262-L264](pkg/kv/kvserver/logstore/logstore.go#L262-L264)
[pkg/storage/pebble_batch.go#L377-L379](pkg/storage/pebble_batch.go#L377-L379)
[external/com_github_cockroachdb_pebble/db.go#L822-L824](external/com_github_cockroachdb_pebble/db.go#L822-L824)
[external/com_github_cockroachdb_pebble/db.go#L874-L876](external/com_github_cockroachdb_pebble/db.go#L874-L876)
[external/com_github_cockroachdb_pebble/commit.go#L316-L318](external/com_github_cockroachdb_pebble/commit.go#L316-L318)
[external/com_github_cockroachdb_pebble/commit.go#L467-L469](external/com_github_cockroachdb_pebble/commit.go#L467-L469)
[external/com_github_cockroachdb_pebble/db.go#L989-L991](external/com_github_cockroachdb_pebble/db.go#L989-L991)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)
[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[external/com_github_cockroachdb_pebble/record/log_writer.go#L584-L586](external/com_github_cockroachdb_pebble/record/log_writer.go#L584-L586)
[GOROOT/src/runtime/pprof/runtime.go#L50-L52](GOROOT/src/runtime/pprof/runtime.go#L50-L52)
[external/com_github_cockroachdb_pebble/record/log_writer.go#L723-L725](external/com_github_cockroachdb_pebble/record/log_writer.go#L723-L725)
[external/com_github_cockroachdb_pebble/record/log_writer.go#L778-L780](external/com_github_cockroachdb_pebble/record/log_writer.go#L778-L780)
[external/com_github_cockroachdb_pebble/record/log_writer.go#L811-L813](external/com_github_cockroachdb_pebble/record/log_writer.go#L811-L813)
[external/com_github_cockroachdb_pebble/vfs/syncing_file.go#L64-L66](external/com_github_cockroachdb_pebble/vfs/syncing_file.go#L64-L66)
```
src/runtime/asm_amd64.s in runtime.goexit at line 1700
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 500
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 319
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 419
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 755
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 875
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1251
pkg/kv/kvserver/replica_raftlog_writes.go in pkg/kv/kvserver.(*replicaLogStorage).appendRaftMuLocked at line 61
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).StoreEntries at line 159
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch at line 263
pkg/storage/pebble_batch.go in pkg/storage.(*writeBatch).CommitNoSyncWait at line 378
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).ApplyNoSyncWait at line 823
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).applyInternal at line 875
external/com_github_cockroachdb_pebble/commit.go in github.com/cockroachdb/pebble.(*commitPipeline).Commit at line 317
external/com_github_cockroachdb_pebble/commit.go in github.com/cockroachdb/pebble.(*commitPipeline).prepare at line 468
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).commitWrite at line 990
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
src/runtime/asm_amd64.s in runtime.goexit at line 1700
external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.NewLogWriter.func2 at line 585
GOROOT/src/runtime/pprof/runtime.go in runtime/pprof.Do at line 51
external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop at line 724
external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushPending at line 779
external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushBlock at line 812
external/com_github_cockroachdb_pebble/vfs/syncing_file.go in github.com/cockroachdb/pebble/vfs.(*syncingFile).Write at line 65
```
### Tags
| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v25.2.4 |
| Go Version | go1.23.7 X:nocoverageredesign |
| Platform | windows amd64 |
| Distribution | CCL |
| Cockroach Release | v25.2.4 |
| Cockroach SHA | db39678e142ec9f40d40b7139c3f821a94fcb80f |
| # of CPUs | 20 |
| # of Goroutines | 601 |
Jira issue: CRDB-67119
Contributor guide
Research direction
Start with pkg/kv/kvserver/logstore/logstore.go and pkg/storage/pebble_batch.go, then trace the reported write failure through the stack trace into Pebble's syncing_file.go. The report only identifies an out-of-disk-space panic and does not define a requested change, reproduction steps, or completion test, so the fix and validation criteria need to be established first.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100