Sentry: store_raft.go:493: log.Fatal: while applying snapshot: while ingesting [× × × × × ×] and excising /Table/26-/Table/27: link × ×: file exists (1) attached stack trace -- stack trace: ...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/6978750266/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/6978750266/?referrer=webhooks_plugin)

Panic Message:

```
store_raft.go:493: log.Fatal: while applying snapshot: while ingesting [× × × × × ×] and excising /Table/26-/Table/27: link × ×: file exists
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:493
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:369
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:433
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1471
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:216
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:337
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:213
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:618
| github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
| github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:165
| google.golang.org/grpc.getChainStreamHandler.func1
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1478
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:178
| google.golang.org/grpc.getChainStreamHandler.func1
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1478
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:145
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:337
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:144
| google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1469
| google.golang.org/grpc.(*Server).processStreamingRPC
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1627
| google.golang.org/grpc.(*Server).handleStream
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1708
| google.golang.org/grpc.(*Server).serveStreams.func1.2
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:965
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (2) secondary error attachment
| while applying snapshot: while ingesting [× × × × × ×] and excising /Table/26-/Table/27: link × ×: file exists
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:986
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:492
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:369
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:433
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1471
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:216
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:337
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:213
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:618
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
| | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
| | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:165
| | google.golang.org/grpc.getChainStreamHandler.func1
| | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1478
| | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
| | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:178
| | google.golang.org/grpc.getChainStreamHandler.func1
| | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1478
| | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
| | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:145
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:337
| | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
| | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:144
| | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
| | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1469
| | google.golang.org/grpc.(*Server).processStreamingRPC
| | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1627
| | google.golang.org/grpc.(*Server).handleStream
| | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1708
| | google.golang.org/grpc.(*Server).serveStreams.func1.2
| | google.golang.org/grpc/external/org_golang_google_grpc/server.go:965
| Wraps: (2) while applying snapshot
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).applySnapshot
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go:686
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:985
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:492
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:369
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:433
| | github.com/coc...
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L964-L966](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L964-L966)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1707-L1709](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1707-L1709)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1626-L1628](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1626-L1628)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1468-L1470](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1468-L1470)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L143-L145](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L143-L145)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L336-L338](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L336-L338)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L144-L146](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L144-L146)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1477-L1479](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1477-L1479)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L177-L179](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L177-L179)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1477-L1479](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1477-L1479)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L164-L166](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L164-L166)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go#L266-L268](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go#L266-L268)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go#L617-L619](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go#L617-L619)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L212-L214](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L212-L214)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L336-L338](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L336-L338)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L215-L217](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L215-L217)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go#L1470-L1472](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go#L1470-L1472)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L432-L434](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L432-L434)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L368-L370](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L368-L370)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L492-L494](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L492-L494)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).serveStreams.func1.2 at line 965
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).handleStream at line 1708
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processStreamingRPC at line 1627
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2 at line 1469
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func2 at line 144
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 337
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func2.1 at line 145
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1478
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func4 at line 178
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1478
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.StreamServerInterceptor.func13 at line 165
pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go in pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler at line 267
pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).RaftSnapshot at line 618
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot at line 213
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 337
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot.func1 at line 216
pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go in pkg/kv/kvserver.(*Store).receiveSnapshot at line 1471
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest at line 433
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).withReplicaForRequest at line 369
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1 at line 493
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v24.1.2 |
| Go Version | go1.22.3 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v24.1.2 |
| Cockroach SHA | 7e81be6de75205c3d08b0d8dcc6ca188306abc27 |
| # of CPUs | 2 |
| # of Goroutines | 266 |

Jira issue: CRDB-55951

Start by tracing snapshot handling from Store.processRaftSnapshotRequest in pkg/kv/kvserver/store_raft.go through Replica.applySnapshot in replica_raftstorage.go and the snapshot receiver in store_snapshot.go. The report provides no reproduction or expected behavior; done would require determining why snapshot ingestion and excision produce a file-exists error and preventing the resulting crash during live-cluster snapshot application.

Written by the indexing model from the issue text.