cockroachdb / cockroachdb/cockroach

Sentry: store_raft.go:557: log.Dev.Fatal: while applying snapshot: while ingesting [× × × × × ×] and excising /System/NodeLivenessMax-/System/tsd: link × ×: operation not permitted (1) attache...

Open
#169,875 1 comment 0 reactions 0 assignees View on GitHub
branch-release-26.1 C-bug O-sentry T-kv X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7464290188/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7464290188/?referrer=webhooks_plugin)

Panic Message:

```
store_raft.go:557: log.Dev.Fatal: while applying snapshot: while ingesting [× × × × × ×] and excising /System/NodeLivenessMax-/System/tsd: link × ×: operation not permitted
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
| pkg/kv/kvserver/store_raft.go:557
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
| pkg/kv/kvserver/store_raft.go:433
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
| pkg/kv/kvserver/store_raft.go:497
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
| pkg/kv/kvserver/store_snapshot.go:602
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
| pkg/kv/kvserver/store_raft.go:283
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| pkg/util/stop/stopper.go:351
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
| pkg/kv/kvserver/store_raft.go:280
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).raftSnapshot
| pkg/kv/kvserver/raft_transport.go:567
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
| pkg/kv/kvserver/raft_transport.go:545
| github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
| bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:123
| google.golang.org/grpc.getChainStreamHandler.func1
| external/org_golang_google_grpc/server.go:1504
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
| pkg/rpc/context.go:190
| google.golang.org/grpc.getChainStreamHandler.func1.getChainStreamHandler.1
| external/org_golang_google_grpc/server.go:1504
| github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor
| pkg/rpc/auth.go:193
| google.golang.org/grpc.getChainStreamHandler.func1
| external/org_golang_google_grpc/server.go:1504
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
| pkg/rpc/context.go:147
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| pkg/util/stop/stopper.go:351
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
| pkg/rpc/context.go:146
| google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
| external/org_golang_google_grpc/server.go:1495
| google.golang.org/grpc.(*Server).processStreamingRPC
| external/org_golang_google_grpc/server.go:1659
| google.golang.org/grpc.(*Server).handleStream
| external/org_golang_google_grpc/server.go:1739
| google.golang.org/grpc.(*Server).serveStreams.func1.1
| external/org_golang_google_grpc/server.go:970
| runtime.goexit
| src/runtime/asm_amd64.s:1693
Wraps: (2) secondary error attachment
| while applying snapshot: while ingesting [× × × × × ×] and excising /System/NodeLivenessMax-/System/tsd: link × ×: operation not permitted
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | pkg/kv/kvserver/replica_raft.go:1161
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
| | pkg/kv/kvserver/store_raft.go:556
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
| | pkg/kv/kvserver/store_raft.go:433
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
| | pkg/kv/kvserver/store_raft.go:497
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
| | pkg/kv/kvserver/store_snapshot.go:602
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
| | pkg/kv/kvserver/store_raft.go:283
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| | pkg/util/stop/stopper.go:351
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
| | pkg/kv/kvserver/store_raft.go:280
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).raftSnapshot
| | pkg/kv/kvserver/raft_transport.go:567
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
| | pkg/kv/kvserver/raft_transport.go:545
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
| | bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
| | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
| | pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:123
| | google.golang.org/grpc.getChainStreamHandler.func1
| | external/org_golang_google_grpc/server.go:1504
| | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
| | pkg/rpc/context.go:190
| | google.golang.org/grpc.getChainStreamHandler.func1.getChainStreamHandler.1
| | external/org_golang_google_grpc/server.go:1504
| | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor
| | pkg/rpc/auth.go:193
| | google.golang.org/grpc.getChainStreamHandler.func1
| | external/org_golang_google_grpc/server.go:1504
| | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
| | pkg/rpc/context.go:147
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| | pkg/util/stop/stopper.go:351
| | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
| | pkg/rpc/context.go:146
| | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
| | external/org_golang_google_grpc/server.go:1495
| | google.golang.org/grpc.(*Server).processStreamingRPC
| | external/org_golang_google_grpc/server.go:1659
| | google.golang.org/grpc.(*Server).handleStream
| | external/org_golang_google_grpc/server.go:1739
| | google.golang.org/grpc.(*Server).serveStreams.func1.1
| | external/org_golang_google_grpc/server.go:970
| Wraps: (2) while applying snapshot
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).applySnapshotRaftMuLocked
| | pkg/kv/kvserver/replica_raftstorage.go:670
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | pkg/kv/kvserver/replica_raft.go:1160
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
| | pkg/kv/kvserver/store_raft.go:556
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
| | pkg/kv/kvserver/store_raft.go:433
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
| | pkg/kv/kvserver/store_raft.go:497
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
| | pkg/kv/kvserver/store_snapshot.go:602
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
| | pkg/kv/kvserver/store_raft.go:283
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| | pkg/util/stop/stopper.go:351
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
| | pkg/kv/kvserver/store_raft.go:280
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).raftSnapshot
| | pkg/kv/kvserver/raft_transport.go:567
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
| | pkg/kv/kvserver/raft_transport.go:545
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
| | bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
| | github.com/c...
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1692-L1694](src/runtime/asm_amd64.s#L1692-L1694)
[external/org_golang_google_grpc/server.go#L969-L971](external/org_golang_google_grpc/server.go#L969-L971)
[external/org_golang_google_grpc/server.go#L1738-L1740](external/org_golang_google_grpc/server.go#L1738-L1740)
[external/org_golang_google_grpc/server.go#L1658-L1660](external/org_golang_google_grpc/server.go#L1658-L1660)
[external/org_golang_google_grpc/server.go#L1494-L1496](external/org_golang_google_grpc/server.go#L1494-L1496)
[pkg/rpc/context.go#L145-L147](pkg/rpc/context.go#L145-L147)
[pkg/util/stop/stopper.go#L350-L352](pkg/util/stop/stopper.go#L350-L352)
[pkg/rpc/context.go#L146-L148](pkg/rpc/context.go#L146-L148)
[external/org_golang_google_grpc/server.go#L1503-L1505](external/org_golang_google_grpc/server.go#L1503-L1505)
[pkg/rpc/auth.go#L192-L194](pkg/rpc/auth.go#L192-L194)
[external/org_golang_google_grpc/server.go#L1503-L1505](external/org_golang_google_grpc/server.go#L1503-L1505)
[pkg/rpc/context.go#L189-L191](pkg/rpc/context.go#L189-L191)
[external/org_golang_google_grpc/server.go#L1503-L1505](external/org_golang_google_grpc/server.go#L1503-L1505)
[pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L122-L124](pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L122-L124)
[bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/https://github.com/cockroachdb/cockroach/blob/ea8d70f63642a8cd69002a2baedc47fa077239cb/pkg/kv/kvserver/storage_services.pb.go#L266-L268](bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/https://github.com/cockroachdb/cockroach/blob/ea8d70f63642a8cd69002a2baedc47fa077239cb/pkg/kv/kvserver/storage_services.pb.go#L266-L268)
[pkg/kv/kvserver/raft_transport.go#L544-L546](pkg/kv/kvserver/raft_transport.go#L544-L546)
[pkg/kv/kvserver/raft_transport.go#L566-L568](pkg/kv/kvserver/raft_transport.go#L566-L568)
[pkg/kv/kvserver/store_raft.go#L279-L281](pkg/kv/kvserver/store_raft.go#L279-L281)
[pkg/util/stop/stopper.go#L350-L352](pkg/util/stop/stopper.go#L350-L352)
[pkg/kv/kvserver/store_raft.go#L282-L284](pkg/kv/kvserver/store_raft.go#L282-L284)
[pkg/kv/kvserver/store_snapshot.go#L601-L603](pkg/kv/kvserver/store_snapshot.go#L601-L603)
[pkg/kv/kvserver/store_raft.go#L496-L498](pkg/kv/kvserver/store_raft.go#L496-L498)
[pkg/kv/kvserver/store_raft.go#L432-L434](pkg/kv/kvserver/store_raft.go#L432-L434)
[pkg/kv/kvserver/store_raft.go#L556-L558](pkg/kv/kvserver/store_raft.go#L556-L558)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1693
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).serveStreams.func1.1 at line 970
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).handleStream at line 1739
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processStreamingRPC at line 1659
external/org_golang_google_grpc/server.go in google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2 at line 1495
pkg/rpc/context.go in pkg/rpc.NewServerEx.func2 at line 146
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 351
pkg/rpc/context.go in pkg/rpc.NewServerEx.func2.1 at line 147
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1504
pkg/rpc/auth.go in pkg/rpc.kvAuth.streamInterceptor at line 193
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1.getChainStreamHandler.1 at line 1504
pkg/rpc/context.go in pkg/rpc.NewServerEx.func4 at line 190
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1504
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.StreamServerInterceptor.func13 at line 123
bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go in pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler at line 267
pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).RaftSnapshot at line 545
pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).raftSnapshot at line 567
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot at line 280
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 351
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot.func1 at line 283
pkg/kv/kvserver/store_snapshot.go in pkg/kv/kvserver.(*Store).receiveSnapshot at line 602
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest at line 497
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).withReplicaForRequest at line 433
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1 at line 557
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v26.1.3 |
| Go Version | go1.25.5 |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v26.1.3 |
| Cockroach SHA | ea8d70f63642a8cd69002a2baedc47fa077239cb |
| # of CPUs | 16 |
| # of Goroutines | 722 |

Jira issue: CRDB-63688

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.