Sentry: replica_proposal.go:346: log.Fatal: lease sequence inversion, prevLease=repl=(n1,s1):1 seq=143 start=1772711676.678561767,0 epo=115 pro=1772711676.678561767,0, newLease=repl=(n3,s3):3 seq=142 ...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7312196119/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7312196119/?referrer=webhooks_plugin)

Panic Message:

```
replica_proposal.go:346: log.Fatal: lease sequence inversion, prevLease=repl=(n1,s1):1 seq=143 start=1772711676.678561767,0 epo=115 pro=1772711676.678561767,0, newLease=repl=(n3,s3):3 seq=142 start=1772053995.136548983,0 epo=128 pro=1772053995.141862688,0
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).leasePostApplyLocked
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:346
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).applySnapshot
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go:820
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:986
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:492
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:369
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:433
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1463
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:216
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:337
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:213
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:618
| github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
| github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:165
| google.golang.org/grpc.getChainStreamHandler.func1
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1478
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:178
| google.golang.org/grpc.getChainStreamHandler.func1
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1478
| github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:157
| google.golang.org/grpc.getChainStreamHandler.func1
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1478
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:145
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:337
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:144
| google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1469
| google.golang.org/grpc.(*Server).processStreamingRPC
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1627
| google.golang.org/grpc.(*Server).handleStream
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1708
| google.golang.org/grpc.(*Server).serveStreams.func1.2
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:965
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (2) log.Fatal: lease sequence inversion, prevLease=repl=(n1,s1):1 seq=143 start=1772711676.678561767,0 epo=115 pro=1772711676.678561767,0, newLease=repl=(n3,s3):3 seq=142 start=1772053995.136548983,0 epo=128 pro=1772053995.141862688,0
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: lease sequence inversion, prevLease=repl=(n1,s1):1 seq=143 start=1772711676.678561767,0 epo=115 pro=1772711676.678561767,0, newLease=repl=(n3,s3):3 seq=142 start=1772053995.136548983,0 epo=128 pro=1772053995.141862688,0
replica_proposal.go:346: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L964-L966](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L964-L966)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1707-L1709](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1707-L1709)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1626-L1628](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1626-L1628)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1468-L1470](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1468-L1470)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/rpc/pkg/rpc/context.go#L143-L145](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/rpc/pkg/rpc/context.go#L143-L145)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/util/stop/stopper.go#L336-L338](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/util/stop/stopper.go#L336-L338)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/rpc/pkg/rpc/context.go#L144-L146](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/rpc/pkg/rpc/context.go#L144-L146)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1477-L1479](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1477-L1479)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/rpc/pkg/rpc/auth.go#L156-L158](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/rpc/pkg/rpc/auth.go#L156-L158)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1477-L1479](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1477-L1479)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/rpc/pkg/rpc/context.go#L177-L179](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/rpc/pkg/rpc/context.go#L177-L179)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1477-L1479](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1477-L1479)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L164-L166](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L164-L166)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go#L266-L268](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go#L266-L268)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go#L617-L619](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go#L617-L619)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L212-L214](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L212-L214)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/util/stop/stopper.go#L336-L338](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/util/stop/stopper.go#L336-L338)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L215-L217](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L215-L217)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go#L1462-L1464](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go#L1462-L1464)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L432-L434](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L432-L434)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L368-L370](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L368-L370)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L491-L493](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L491-L493)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L985-L987](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L985-L987)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go#L819-L821](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go#L819-L821)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L345-L347](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L345-L347)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).serveStreams.func1.2 at line 965
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).handleStream at line 1708
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processStreamingRPC at line 1627
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2 at line 1469
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func2 at line 144
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 337
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func2.1 at line 145
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1478
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.streamInterceptor at line 157
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1478
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func4 at line 178
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1478
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.StreamServerInterceptor.func13 at line 165
pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go in pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler at line 267
pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).RaftSnapshot at line 618
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot at line 213
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 337
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot.func1 at line 216
pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go in pkg/kv/kvserver.(*Store).receiveSnapshot at line 1463
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest at line 433
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).withReplicaForRequest at line 369
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1 at line 492
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 986
pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go in pkg/kv/kvserver.(*Replica).applySnapshot at line 820
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).leasePostApplyLocked at line 346
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v24.1.3 |
| Go Version | go1.22.5 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v24.1.3 |
| Cockroach SHA | 9e10212477fde97f55ea4fff01797288c836575c |
| # of CPUs | 2 |
| # of Goroutines | 351 |

Jira issue: CRDB-61074