Sentry: store_raft.go:486: log.Fatal: while applying snapshot: error rate limiting bulk io write: rate: Wait(n=909) exceeds limiter's burst 228 (1) attached stack trace -- stack trace: | github.co...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/6964870757/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/6964870757/?referrer=webhooks_plugin)

Panic Message:

```
store_raft.go:486: log.Fatal: while applying snapshot: error rate limiting bulk io write: rate: Wait(n=909) exceeds limiter's burst 228
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
| pkg/kv/kvserver/store_raft.go:486
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
| pkg/kv/kvserver/store_raft.go:362
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
| pkg/kv/kvserver/store_raft.go:426
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
| pkg/kv/kvserver/store_snapshot.go:1705
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
| pkg/kv/kvserver/store_raft.go:209
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| pkg/util/stop/stopper.go:349
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
| pkg/kv/kvserver/store_raft.go:206
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
| pkg/kv/kvserver/raft_transport.go:638
| github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
| bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160
| google.golang.org/grpc.getChainStreamHandler.func1
| external/org_golang_google_grpc/server.go:1481
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
| pkg/rpc/context.go:174
| google.golang.org/grpc.getChainStreamHandler.func1
| external/org_golang_google_grpc/server.go:1481
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
| pkg/rpc/context.go:141
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| pkg/util/stop/stopper.go:349
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
| pkg/rpc/context.go:140
| google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
| external/org_golang_google_grpc/server.go:1472
| google.golang.org/grpc.(*Server).processStreamingRPC
| external/org_golang_google_grpc/server.go:1636
| google.golang.org/grpc.(*Server).handleStream
| external/org_golang_google_grpc/server.go:1716
| google.golang.org/grpc.(*Server).serveStreams.func1.1
| external/org_golang_google_grpc/server.go:947
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (2) secondary error attachment
| while applying snapshot: error rate limiting bulk io write: rate: Wait(n=909) exceeds limiter's burst 228
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | pkg/kv/kvserver/replica_raft.go:1152
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
| | pkg/kv/kvserver/store_raft.go:485
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
| | pkg/kv/kvserver/store_raft.go:362
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
| | pkg/kv/kvserver/store_raft.go:426
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
| | pkg/kv/kvserver/store_snapshot.go:1705
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
| | pkg/kv/kvserver/store_raft.go:209
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| | pkg/util/stop/stopper.go:349
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
| | pkg/kv/kvserver/store_raft.go:206
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
| | pkg/kv/kvserver/raft_transport.go:638
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
| | bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
| | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
| | pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160
| | google.golang.org/grpc.getChainStreamHandler.func1
| | external/org_golang_google_grpc/server.go:1481
| | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
| | pkg/rpc/context.go:174
| | google.golang.org/grpc.getChainStreamHandler.func1
| | external/org_golang_google_grpc/server.go:1481
| | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
| | pkg/rpc/context.go:141
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| | pkg/util/stop/stopper.go:349
| | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
| | pkg/rpc/context.go:140
| | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
| | external/org_golang_google_grpc/server.go:1472
| | google.golang.org/grpc.(*Server).processStreamingRPC
| | external/org_golang_google_grpc/server.go:1636
| | google.golang.org/grpc.(*Server).handleStream
| | external/org_golang_google_grpc/server.go:1716
| | google.golang.org/grpc.(*Server).serveStreams.func1.1
| | external/org_golang_google_grpc/server.go:947
| Wraps: (2) while applying snapshot
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvserverbase.LimitBulkIOWrite
| | pkg/kv/kvserver/kvserverbase/syncing_write.go:51
| | [...repeated from below...]
| Wraps: (4) error rate limiting bulk io write
| Wraps: (5) attached stack trace
| -- stack trace:
| | golang.org/x/time/rate.(*Limiter).wait
| | external/org_golang_x_time/rate/rate.go:258
| | golang.org/x/time/rate.(*Limiter).WaitN
| | external/org_golang_x_time/rate/rate.go:247
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvserverbase.LimitBulkIOWrite
| | pkg/kv/kvserver/kvserverbase/syncing_write.go:50
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*SSTSnapshotStorageFile).Write
| | pkg/kv/kvserver/replica_sst_snapshot_storage.go:229
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*SSTSnapshotStorageScratch).WriteSST
| | pkg/kv/kvserver/replica_sst_snapshot_storage.go:155
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).applySnapshot
| | pkg/kv/kvserver/replica_raftstorage.go:534
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | pkg/kv/kvserver/replica_raft.go:1151
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
| | pkg/kv/kvserver/store_raft.go:485
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
| | pkg/kv/kvserver/store_raft.go:362
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
| | pkg/kv/kvserver/store_raft.go:426
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
| | pkg/kv/kvserver/store_snapshot.go:1705
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
| | pkg/kv/kvserver/store_raft.go:209
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| | pkg/util/stop/stopper.go:349
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
| | pkg/kv/kvserver/store_raft.go:206
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
| | pkg/kv/kvserver/raft_transport.go:638
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
| | bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
| | github.com/co...
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[external/org_golang_google_grpc/server.go#L946-L948](external/org_golang_google_grpc/server.go#L946-L948)
[external/org_golang_google_grpc/server.go#L1715-L1717](external/org_golang_google_grpc/server.go#L1715-L1717)
[external/org_golang_google_grpc/server.go#L1635-L1637](external/org_golang_google_grpc/server.go#L1635-L1637)
[external/org_golang_google_grpc/server.go#L1471-L1473](external/org_golang_google_grpc/server.go#L1471-L1473)
[pkg/rpc/context.go#L139-L141](pkg/rpc/context.go#L139-L141)
[pkg/util/stop/stopper.go#L348-L350](pkg/util/stop/stopper.go#L348-L350)
[pkg/rpc/context.go#L140-L142](pkg/rpc/context.go#L140-L142)
[external/org_golang_google_grpc/server.go#L1480-L1482](external/org_golang_google_grpc/server.go#L1480-L1482)
[pkg/rpc/context.go#L173-L175](pkg/rpc/context.go#L173-L175)
[external/org_golang_google_grpc/server.go#L1480-L1482](external/org_golang_google_grpc/server.go#L1480-L1482)
[pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L159-L161](pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L159-L161)
[bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/https://github.com/cockroachdb/cockroach/blob/ef2ebe96f3a26fd6d6716e2ec4a0c8c6fe0fb567/pkg/kv/kvserver/storage_services.pb.go#L266-L268](bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/https://github.com/cockroachdb/cockroach/blob/ef2ebe96f3a26fd6d6716e2ec4a0c8c6fe0fb567/pkg/kv/kvserver/storage_services.pb.go#L266-L268)
[pkg/kv/kvserver/raft_transport.go#L637-L639](pkg/kv/kvserver/raft_transport.go#L637-L639)
[pkg/kv/kvserver/store_raft.go#L205-L207](pkg/kv/kvserver/store_raft.go#L205-L207)
[pkg/util/stop/stopper.go#L348-L350](pkg/util/stop/stopper.go#L348-L350)
[pkg/kv/kvserver/store_raft.go#L208-L210](pkg/kv/kvserver/store_raft.go#L208-L210)
[pkg/kv/kvserver/store_snapshot.go#L1704-L1706](pkg/kv/kvserver/store_snapshot.go#L1704-L1706)
[pkg/kv/kvserver/store_raft.go#L425-L427](pkg/kv/kvserver/store_raft.go#L425-L427)
[pkg/kv/kvserver/store_raft.go#L361-L363](pkg/kv/kvserver/store_raft.go#L361-L363)
[pkg/kv/kvserver/store_raft.go#L485-L487](pkg/kv/kvserver/store_raft.go#L485-L487)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).serveStreams.func1.1 at line 947
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).handleStream at line 1716
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processStreamingRPC at line 1636
external/org_golang_google_grpc/server.go in google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2 at line 1472
pkg/rpc/context.go in pkg/rpc.NewServerEx.func2 at line 140
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 349
pkg/rpc/context.go in pkg/rpc.NewServerEx.func2.1 at line 141
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1481
pkg/rpc/context.go in pkg/rpc.NewServerEx.func4 at line 174
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1481
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.StreamServerInterceptor.func13 at line 160
bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go in pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler at line 267
pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).RaftSnapshot at line 638
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot at line 206
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 349
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot.func1 at line 209
pkg/kv/kvserver/store_snapshot.go in pkg/kv/kvserver.(*Store).receiveSnapshot at line 1705
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest at line 426
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).withReplicaForRequest at line 362
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1 at line 486
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v24.3.0 |
| Go Version | go1.22.8 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v24.3.0 |
| Cockroach SHA | ef2ebe96f3a26fd6d6716e2ec4a0c8c6fe0fb567 |
| # of CPUs | 8 |
| # of Goroutines | 375 |

Jira issue: CRDB-55638