cockroachdb / cockroachdb/cockroach
Sentry: filenames.go:302: log.Dev.Fatal: rename × ×: The system cannot find the file specified. (1) attached stack trace -- stack trace: | github.com/cockroachdb/pebble/internal/base.MustExist ...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/7581870418/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7581870418/?referrer=webhooks_plugin)
Panic Message:
```
filenames.go:302: log.Dev.Fatal: rename × ×: The system cannot find the file specified.
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/pebble/internal/base.MustExist
| external/com_github_cockroachdb_pebble/internal/base/filenames.go:302
| github.com/cockroachdb/pebble/wal.(*StandaloneManager).Create
| external/com_github_cockroachdb_pebble/wal/standalone_manager.go:161
| github.com/cockroachdb/pebble.(*DB).rotateWAL
| external/com_github_cockroachdb_pebble/db.go:2650
| github.com/cockroachdb/pebble.(*DB).makeRoomForWrite
| external/com_github_cockroachdb_pebble/db.go:2502
| github.com/cockroachdb/pebble.(*DB).commitWrite.func1
| external/com_github_cockroachdb_pebble/db.go:934
| github.com/cockroachdb/pebble.(*DB).commitWrite
| external/com_github_cockroachdb_pebble/db.go:936
| github.com/cockroachdb/pebble.(*commitPipeline).prepare
| external/com_github_cockroachdb_pebble/commit.go:468
| github.com/cockroachdb/pebble.(*commitPipeline).Commit
| external/com_github_cockroachdb_pebble/commit.go:317
| github.com/cockroachdb/pebble.(*DB).applyInternal
| external/com_github_cockroachdb_pebble/db.go:835
| github.com/cockroachdb/pebble.(*DB).ApplyNoSyncWait
| external/com_github_cockroachdb_pebble/db.go:783
| github.com/cockroachdb/cockroach/pkg/storage.(*writeBatch).CommitNoSyncWait
| pkg/storage/pebble_batch.go:383
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch
| pkg/kv/kvserver/logstore/logstore.go:272
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).StoreEntries
| pkg/kv/kvserver/logstore/logstore.go:171
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaLogStorage).appendRaftMuLocked
| pkg/kv/kvserver/replica_raftlog_writes.go:61
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1219
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| pkg/kv/kvserver/replica_raft.go:884
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:760
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:421
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:314
| runtime.goexit
| src/runtime/asm_amd64.s:1693
Wraps: (2) secondary error attachment
| rename × ×: The system cannot find the file specified.
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/vfs.defaultFS.ReuseForWrite
| | external/com_github_cockroachdb_pebble/vfs/vfs.go:314
| | [...repeated from below...]
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/vfs.defaultFS.Rename
| | external/com_github_cockroachdb_pebble/vfs/vfs.go:307
| | github.com/cockroachdb/pebble/vfs.defaultFS.ReuseForWrite
| | external/com_github_cockroachdb_pebble/vfs/vfs.go:313
| | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).ReuseForWrite.func1
| | external/com_github_cockroachdb_pebble/vfs/disk_health.go:893
| | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).timeFilesystemOp
| | external/com_github_cockroachdb_pebble/vfs/disk_health.go:665
| | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).ReuseForWrite
| | external/com_github_cockroachdb_pebble/vfs/disk_health.go:892
| | github.com/cockroachdb/pebble/vfs.(*enospcFS).ReuseForWrite
| | external/com_github_cockroachdb_pebble/vfs/disk_full.go:296
| | github.com/cockroachdb/cockroach/pkg/storage/fs.(*Env).ReuseForWrite
| | pkg/storage/fs/fs.go:442
| | github.com/cockroachdb/pebble/wal.(*StandaloneManager).Create
| | external/com_github_cockroachdb_pebble/wal/standalone_manager.go:157
| | github.com/cockroachdb/pebble.(*DB).rotateWAL
| | external/com_github_cockroachdb_pebble/db.go:2650
| | github.com/cockroachdb/pebble.(*DB).makeRoomForWrite
| | external/com_github_cockroachdb_pebble/db.go:2502
| | github.com/cockroachdb/pebble.(*DB).commitWrite.func1
| | external/com_github_cockroachdb_pebble/db.go:934
| | github.com/cockroachdb/pebble.(*DB).commitWrite
| | external/com_github_cockroachdb_pebble/db.go:936
| | github.com/cockroachdb/pebble.(*commitPipeline).prepare
| | external/com_github_cockroachdb_pebble/commit.go:468
| | github.com/cockroachdb/pebble.(*commitPipeline).Commit
| | external/com_github_cockroachdb_pebble/commit.go:317
| | github.com/cockroachdb/pebble.(*DB).applyInternal
| | external/com_github_cockroachdb_pebble/db.go:835
| | github.com/cockroachdb/pebble.(*DB).ApplyNoSyncWait
| | external/com_github_cockroachdb_pebble/db.go:783
| | github.com/cockroachdb/cockroach/pkg/storage.(*writeBatch).CommitNoSyncWait
| | pkg/storage/pebble_batch.go:383
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch
| | pkg/kv/kvserver/logstore/logstore.go:272
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).StoreEntries
| | pkg/kv/kvserver/logstore/logstore.go:171
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaLogStorage).appendRaftMuLocked
| | pkg/kv/kvserver/replica_raftlog_writes.go:61
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | pkg/kv/kvserver/replica_raft.go:1219
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| | pkg/kv/kvserver/replica_raft.go:884
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| | pkg/kv/kvserver/store_raft.go:760
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | pkg/kv/kvserver/scheduler.go:421
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| | pkg/kv/kvserver/scheduler.go:314
| | runtime.goexit
| | src/runtime/asm_amd64.s:1693
| Wraps: (4) rename × ×
| Wraps: (5) The system cannot find the file specified.
| Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *withstack.withStack (4) *os.LinkError (5) syscall.Errno
Wraps: (3) log.Dev.Fatal: rename × ×: The system cannot find the file specified.
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/vfs.defaultFS.ReuseForWrite
| | external/com_github_cockroachdb_pebble/vfs/vfs.go:314
| | [...repeated from below...]
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/vfs.defaultFS.Rename
| | external/com_github_cockroachdb_pebble/vfs/vfs.go:307
| | github.com/cockroachdb/pebble/vfs.defaultFS.ReuseForWrite
| | external/com_github_cockroachdb_pebble/vfs/vfs.go:313
| | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).ReuseForWrite.func1
| | external/com_github_cockroachdb_pebble/vfs/disk_health.go:893
| | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).timeFilesystemOp
| | external/com_github_cockroachdb_pebble/vfs/disk_health.go:665
| | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).ReuseForWrite
| | external/com_github_cockroachdb_pebble/vfs/disk_health.go:892
| | github.com/cockroachdb/pebble/vfs.(*enospcFS).ReuseForWrite
| | external/com_github_cockroachdb_pebble/vfs/disk_full.go:296
| | github.com/cockroachdb/cockroach/pkg/storage/fs.(*Env).ReuseForWrite
| | pkg/storage/fs/fs.go:442
| | github.com/cockroachdb/pebble/wal.(*StandaloneManager).Create
| | external/com_github_cockroachdb_pebble/wal/standalone_manager.go:157
| | github.com/cockroachdb/pebble.(*DB).rotateWAL
| | external/com_github_cockroachdb_pebble/db.go:2650
| | github.com/cockroachdb/pebble.(*DB).makeRoomForWrite
| | external/com_github_cockroachdb_pebble/db.go:2502
| | github.com/cockroachdb/pebbl...
```
Stacktrace (expand for inline code snippets):
[src/runtime/asm_amd64.s#L1692-L1694](src/runtime/asm_amd64.s#L1692-L1694)
[pkg/kv/kvserver/scheduler.go#L313-L315](pkg/kv/kvserver/scheduler.go#L313-L315)
[pkg/kv/kvserver/scheduler.go#L420-L422](pkg/kv/kvserver/scheduler.go#L420-L422)
[pkg/kv/kvserver/store_raft.go#L759-L761](pkg/kv/kvserver/store_raft.go#L759-L761)
[pkg/kv/kvserver/replica_raft.go#L883-L885](pkg/kv/kvserver/replica_raft.go#L883-L885)
[pkg/kv/kvserver/replica_raft.go#L1218-L1220](pkg/kv/kvserver/replica_raft.go#L1218-L1220)
[pkg/kv/kvserver/replica_raftlog_writes.go#L60-L62](pkg/kv/kvserver/replica_raftlog_writes.go#L60-L62)
[pkg/kv/kvserver/logstore/logstore.go#L170-L172](pkg/kv/kvserver/logstore/logstore.go#L170-L172)
[pkg/kv/kvserver/logstore/logstore.go#L271-L273](pkg/kv/kvserver/logstore/logstore.go#L271-L273)
[pkg/storage/pebble_batch.go#L382-L384](pkg/storage/pebble_batch.go#L382-L384)
[external/com_github_cockroachdb_pebble/db.go#L782-L784](external/com_github_cockroachdb_pebble/db.go#L782-L784)
[external/com_github_cockroachdb_pebble/db.go#L834-L836](external/com_github_cockroachdb_pebble/db.go#L834-L836)
[external/com_github_cockroachdb_pebble/commit.go#L316-L318](external/com_github_cockroachdb_pebble/commit.go#L316-L318)
[external/com_github_cockroachdb_pebble/commit.go#L467-L469](external/com_github_cockroachdb_pebble/commit.go#L467-L469)
[external/com_github_cockroachdb_pebble/db.go#L935-L937](external/com_github_cockroachdb_pebble/db.go#L935-L937)
[external/com_github_cockroachdb_pebble/db.go#L933-L935](external/com_github_cockroachdb_pebble/db.go#L933-L935)
[external/com_github_cockroachdb_pebble/db.go#L2501-L2503](external/com_github_cockroachdb_pebble/db.go#L2501-L2503)
[external/com_github_cockroachdb_pebble/db.go#L2649-L2651](external/com_github_cockroachdb_pebble/db.go#L2649-L2651)
[external/com_github_cockroachdb_pebble/wal/standalone_manager.go#L160-L162](external/com_github_cockroachdb_pebble/wal/standalone_manager.go#L160-L162)
[external/com_github_cockroachdb_pebble/internal/base/filenames.go#L301-L303](external/com_github_cockroachdb_pebble/internal/base/filenames.go#L301-L303)
```
src/runtime/asm_amd64.s in runtime.goexit at line 1693
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 314
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 421
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 760
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 884
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1219
pkg/kv/kvserver/replica_raftlog_writes.go in pkg/kv/kvserver.(*replicaLogStorage).appendRaftMuLocked at line 61
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).StoreEntries at line 171
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch at line 272
pkg/storage/pebble_batch.go in pkg/storage.(*writeBatch).CommitNoSyncWait at line 383
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).ApplyNoSyncWait at line 783
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).applyInternal at line 835
external/com_github_cockroachdb_pebble/commit.go in github.com/cockroachdb/pebble.(*commitPipeline).Commit at line 317
external/com_github_cockroachdb_pebble/commit.go in github.com/cockroachdb/pebble.(*commitPipeline).prepare at line 468
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).commitWrite at line 936
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).commitWrite.func1 at line 934
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).makeRoomForWrite at line 2502
external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).rotateWAL at line 2650
external/com_github_cockroachdb_pebble/wal/standalone_manager.go in github.com/cockroachdb/pebble/wal.(*StandaloneManager).Create at line 161
external/com_github_cockroachdb_pebble/internal/base/filenames.go in github.com/cockroachdb/pebble/internal/base.MustExist at line 302
```
### Tags
| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v26.2.1 |
| Go Version | go1.25.5 |
| Platform | windows amd64 |
| Distribution | CCL |
| Cockroach Release | v26.2.1 |
| Cockroach SHA | b398a6e1fde74a915daf8e9841303c8ee39c2abe |
| # of CPUs | 4 |
| # of Goroutines | 346 |
Jira issue: CRDB-65222
Contributor guide
Research direction
Start with external/com_github_cockroachdb_pebble/wal/standalone_manager.go:157-161 and follow ReuseForWrite through pkg/storage/fs/fs.go:442 and the Pebble VFS paths in the stack trace. Trace how WAL rotation reaches filenames.go:302 and determine why the rename reports a missing file. Done means the reported crash is explained and a validated fix prevents this failure path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100