cockroachdb / cockroachdb/cockroach

Sentry: syncing_file.go:65: write ×: There is not enough space on the disk. (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:791 | github.com/cockroac...

Open
#173,819 1 comment 0 reactions 0 assignees View on GitHub
A-storage branch-release-25.2 C-bug O-sentry T-storage X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7689462230/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7689462230/?referrer=webhooks_plugin)

Panic Message:

```
syncing_file.go:65: write ×: There is not enough space on the disk.
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| github.com/cockroachdb/pebble.(*DB).commitWrite
| external/com_github_cockroachdb_pebble/db.go:990
| github.com/cockroachdb/pebble.(*commitPipeline).prepare
| external/com_github_cockroachdb_pebble/commit.go:468
| github.com/cockroachdb/pebble.(*commitPipeline).Commit
| external/com_github_cockroachdb_pebble/commit.go:317
| github.com/cockroachdb/pebble.(*DB).applyInternal
| external/com_github_cockroachdb_pebble/db.go:875
| github.com/cockroachdb/pebble.(*DB).ApplyNoSyncWait
| external/com_github_cockroachdb_pebble/db.go:823
| github.com/cockroachdb/cockroach/pkg/storage.(*writeBatch).CommitNoSyncWait
| pkg/storage/pebble_batch.go:378
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch
| pkg/kv/kvserver/logstore/logstore.go:263
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).StoreEntries
| pkg/kv/kvserver/logstore/logstore.go:159
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaLogStorage).appendRaftMuLocked
| pkg/kv/kvserver/replica_raftlog_writes.go:61
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1251
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| pkg/kv/kvserver/replica_raft.go:875
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:755
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:419
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:319
| [...repeated from below...]
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/pebble/vfs.(*syncingFile).Write
| external/com_github_cockroachdb_pebble/vfs/syncing_file.go:65
| github.com/cockroachdb/pebble/record.(*LogWriter).flushBlock
| external/com_github_cockroachdb_pebble/record/log_writer.go:812
| github.com/cockroachdb/pebble/record.(*LogWriter).flushPending
| external/com_github_cockroachdb_pebble/record/log_writer.go:779
| github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop
| external/com_github_cockroachdb_pebble/record/log_writer.go:724
| runtime/pprof.Do
| GOROOT/src/runtime/pprof/runtime.go:51
| github.com/cockroachdb/pebble/record.NewLogWriter.func2
| external/com_github_cockroachdb_pebble/record/log_writer.go:585
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (3) write ×
Wraps: (4) There is not enough space on the disk.
Error types: (1) *withstack.withStack (2) *withstack.withStack (3) *fs.PathError (4) syscall.Errno
-- report composition:
syscall.Errno
*fs.PathError
syncing_file.go:65: *withstack.withStack (top exception)
panic.go:791: *withstack.withStack (1)
(check the extra data payloads)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[pkg/util/stop/stopper.go#L499-L501](pkg/util/stop/stopper.go#L499-L501)
[pkg/kv/kvserver/scheduler.go#L318-L320](pkg/kv/kvserver/scheduler.go#L318-L320)
[pkg/kv/kvserver/scheduler.go#L418-L420](pkg/kv/kvserver/scheduler.go#L418-L420)
[pkg/kv/kvserver/store_raft.go#L754-L756](pkg/kv/kvserver/store_raft.go#L754-L756)
[pkg/kv/kvserver/replica_raft.go#L874-L876](pkg/kv/kvserver/replica_raft.go#L874-L876)
[pkg/kv/kvserver/replica_raft.go#L1250-L1252](pkg/kv/kvserver/replica_raft.go#L1250-L1252)
[pkg/kv/kvserver/replica_raftlog_writes.go#L60-L62](pkg/kv/kvserver/replica_raftlog_writes.go#L60-L62)
[pkg/kv/kvserver/logstore/logstore.go#L158-L160](pkg/kv/kvserver/logstore/logstore.go#L158-L160)
[pkg/kv/kvserver/logstore/logstore.go#L262-L264](pkg/kv/kvserver/logstore/logstore.go#L262-L264)
[pkg/storage/pebble_batch.go#L377-L379](pkg/storage/pebble_batch.go#L377-L379)
[external/com_github_cockroachdb_pebble/db.go#L822-L824](external/com_github_cockroachdb_pebble/db.go#L822-L824)
[external/com_github_cockroachdb_pebble/db.go#L874-L876](external/com_github_cockroachdb_pebble/db.go#L874-L876)
[external/com_github_cockroachdb_pebble/commit.go#L316-L318](external/com_github_cockroachdb_pebble/commit.go#L316-L318)
[external/com_github_cockroachdb_pebble/commit.go#L467-L469](external/com_github_cockroachdb_pebble/commit.go#L467-L469)
[external/com_github_cockroachdb_pebble/db.go#L989-L991](external/com_github_cockroachdb_pebble/db.go#L989-L991)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)
[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[external/com_github_cockroachdb_pebble/record/log_writer.go#L584-L586](external/com_github_cockroachdb_pebble/record/log_writer.go#L584-L586)
[GOROOT/src/runtime/pprof/runtime.go#L50-L52](GOROOT/src/runtime/pprof/runtime.go#L50-L52)
[external/com_github_cockroachdb_pebble/record/log_writer.go#L723-L725](external/com_github_cockroachdb_pebble/record/log_writer.go#L723-L725)
[external/com_github_cockroachdb_pebble/record/log_writer.go#L778-L780](external/com_github_cockroachdb_pebble/record/log_writer.go#L778-L780)
[external/com_github_cockroachdb_pebble/record/log_writer.go#L811-L813](external/com_github_cockroachdb_pebble/record/log_writer.go#L811-L813)
[external/com_github_cockroachdb_pebble/vfs/syncing_file.go#L64-L66](external/com_github_cockroachdb_pebble/vfs/syncing_file.go#L64-L66)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1700
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 500
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 319
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 419
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 755
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 875
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1251
pkg/kv/kvserver/replica_raftlog_writes.go in pkg/kv/kvserver.(*replicaLogStorage).appendRaftMuLocked at line 61
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).StoreEntries at line 159
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch at line 263
pkg/storage/pebble_batch.go in pkg/storage.(*writeBatch).CommitNoSyncWait at line 378
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).ApplyNoSyncWait at line 823
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).applyInternal at line 875
external/com_github_cockroachdb_pebble/commit.go in github.com/cockroachdb/pebble.(*commitPipeline).Commit at line 317
external/com_github_cockroachdb_pebble/commit.go in github.com/cockroachdb/pebble.(*commitPipeline).prepare at line 468
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).commitWrite at line 990
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
src/runtime/asm_amd64.s in runtime.goexit at line 1700
external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.NewLogWriter.func2 at line 585
GOROOT/src/runtime/pprof/runtime.go in runtime/pprof.Do at line 51
external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop at line 724
external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushPending at line 779
external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushBlock at line 812
external/com_github_cockroachdb_pebble/vfs/syncing_file.go in github.com/cockroachdb/pebble/vfs.(*syncingFile).Write at line 65
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v25.2.4 |
| Go Version | go1.23.7 X:nocoverageredesign |
| Platform | windows amd64 |
| Distribution | CCL |
| Cockroach Release | v25.2.4 |
| Cockroach SHA | db39678e142ec9f40d40b7139c3f821a94fcb80f |
| # of CPUs | 20 |
| # of Goroutines | 601 |

Jira issue: CRDB-67119

Contributor guide

Open the contributing guide

Research direction

Start with pkg/kv/kvserver/logstore/logstore.go and pkg/storage/pebble_batch.go, then trace the reported write failure through the stack trace into Pebble's syncing_file.go. The report only identifies an out-of-disk-space panic and does not define a requested change, reproduction steps, or completion test, so the fix and validation criteria need to be established first.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.