Sentry: replica_proposal.go:362: log.Fatal: lease sequence inversion, prevLease=repl=(n1,s1):1 seq=6 start=1763761633.550170071,0 term=13 min-exp=1763761636.716094473,0 pro=1763761633.716094473,0 acq=...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7052459057/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7052459057/?referrer=webhooks_plugin)

Panic Message:

```
replica_proposal.go:362: log.Fatal: lease sequence inversion, prevLease=repl=(n1,s1):1 seq=6 start=1763761633.550170071,0 term=13 min-exp=1763761636.716094473,0 pro=1763761633.716094473,0 acq=Request, newLease=repl=(n2,s2):3 seq=4 start=1763761482.907503188,0 term=14 min-exp=1763761488.910356849,0 pro=1763761482.965276347,0 acq=Request
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).leasePostApplyLocked
| pkg/kv/kvserver/replica_proposal.go:362
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).applySnapshotRaftMuLocked
| pkg/kv/kvserver/replica_raftstorage.go:758
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1155
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
| pkg/kv/kvserver/store_raft.go:558
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
| pkg/kv/kvserver/store_raft.go:435
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
| pkg/kv/kvserver/store_raft.go:499
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
| pkg/kv/kvserver/store_snapshot.go:600
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
| pkg/kv/kvserver/store_raft.go:282
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| pkg/util/stop/stopper.go:349
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
| pkg/kv/kvserver/store_raft.go:279
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).raftSnapshot
| pkg/kv/kvserver/raft_transport.go:566
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
| pkg/kv/kvserver/raft_transport.go:544
| github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
| bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:159
| google.golang.org/grpc.getChainStreamHandler.func1
| external/org_golang_google_grpc/server.go:1504
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
| pkg/rpc/context.go:187
| google.golang.org/grpc.getChainStreamHandler.func1
| external/org_golang_google_grpc/server.go:1504
| github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor
| pkg/rpc/auth.go:152
| google.golang.org/grpc.getChainStreamHandler.func1
| external/org_golang_google_grpc/server.go:1504
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
| pkg/rpc/context.go:145
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| pkg/util/stop/stopper.go:349
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
| pkg/rpc/context.go:144
| google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
| external/org_golang_google_grpc/server.go:1495
| google.golang.org/grpc.(*Server).processStreamingRPC
| external/org_golang_google_grpc/server.go:1659
| google.golang.org/grpc.(*Server).handleStream
| external/org_golang_google_grpc/server.go:1739
| google.golang.org/grpc.(*Server).serveStreams.func1.1
| external/org_golang_google_grpc/server.go:970
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (2) log.Fatal: lease sequence inversion, prevLease=repl=(n1,s1):1 seq=6 start=1763761633.550170071,0 term=13 min-exp=1763761636.716094473,0 pro=1763761633.716094473,0 acq=Request, newLease=repl=(n2,s2):3 seq=4 start=1763761482.907503188,0 term=14 min-exp=1763761488.910356849,0 pro=1763761482.965276347,0 acq=Request
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: lease sequence inversion, prevLease=repl=(n1,s1):1 seq=6 start=1763761633.550170071,0 term=13 min-exp=1763761636.716094473,0 pro=1763761633.716094473,0 acq=Request, newLease=repl=(n2,s2):3 seq=4 start=1763761482.907503188,0 term=14 min-exp=1763761488.910356849,0 pro=1763761482.965276347,0 acq=Request
replica_proposal.go:362: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[external/org_golang_google_grpc/server.go#L969-L971](external/org_golang_google_grpc/server.go#L969-L971)
[external/org_golang_google_grpc/server.go#L1738-L1740](external/org_golang_google_grpc/server.go#L1738-L1740)
[external/org_golang_google_grpc/server.go#L1658-L1660](external/org_golang_google_grpc/server.go#L1658-L1660)
[external/org_golang_google_grpc/server.go#L1494-L1496](external/org_golang_google_grpc/server.go#L1494-L1496)
[pkg/rpc/context.go#L143-L145](pkg/rpc/context.go#L143-L145)
[pkg/util/stop/stopper.go#L348-L350](pkg/util/stop/stopper.go#L348-L350)
[pkg/rpc/context.go#L144-L146](pkg/rpc/context.go#L144-L146)
[external/org_golang_google_grpc/server.go#L1503-L1505](external/org_golang_google_grpc/server.go#L1503-L1505)
[pkg/rpc/auth.go#L151-L153](pkg/rpc/auth.go#L151-L153)
[external/org_golang_google_grpc/server.go#L1503-L1505](external/org_golang_google_grpc/server.go#L1503-L1505)
[pkg/rpc/context.go#L186-L188](pkg/rpc/context.go#L186-L188)
[external/org_golang_google_grpc/server.go#L1503-L1505](external/org_golang_google_grpc/server.go#L1503-L1505)
[pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L158-L160](pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L158-L160)
[bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/https://github.com/cockroachdb/cockroach/blob/1cddb3a22695e967197844b214115eb645473efe/pkg/kv/kvserver/storage_services.pb.go#L266-L268](bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/https://github.com/cockroachdb/cockroach/blob/1cddb3a22695e967197844b214115eb645473efe/pkg/kv/kvserver/storage_services.pb.go#L266-L268)
[pkg/kv/kvserver/raft_transport.go#L543-L545](pkg/kv/kvserver/raft_transport.go#L543-L545)
[pkg/kv/kvserver/raft_transport.go#L565-L567](pkg/kv/kvserver/raft_transport.go#L565-L567)
[pkg/kv/kvserver/store_raft.go#L278-L280](pkg/kv/kvserver/store_raft.go#L278-L280)
[pkg/util/stop/stopper.go#L348-L350](pkg/util/stop/stopper.go#L348-L350)
[pkg/kv/kvserver/store_raft.go#L281-L283](pkg/kv/kvserver/store_raft.go#L281-L283)
[pkg/kv/kvserver/store_snapshot.go#L599-L601](pkg/kv/kvserver/store_snapshot.go#L599-L601)
[pkg/kv/kvserver/store_raft.go#L498-L500](pkg/kv/kvserver/store_raft.go#L498-L500)
[pkg/kv/kvserver/store_raft.go#L434-L436](pkg/kv/kvserver/store_raft.go#L434-L436)
[pkg/kv/kvserver/store_raft.go#L557-L559](pkg/kv/kvserver/store_raft.go#L557-L559)
[pkg/kv/kvserver/replica_raft.go#L1154-L1156](pkg/kv/kvserver/replica_raft.go#L1154-L1156)
[pkg/kv/kvserver/replica_raftstorage.go#L757-L759](pkg/kv/kvserver/replica_raftstorage.go#L757-L759)
[pkg/kv/kvserver/replica_proposal.go#L361-L363](pkg/kv/kvserver/replica_proposal.go#L361-L363)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1700
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).serveStreams.func1.1 at line 970
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).handleStream at line 1739
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processStreamingRPC at line 1659
external/org_golang_google_grpc/server.go in google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2 at line 1495
pkg/rpc/context.go in pkg/rpc.NewServerEx.func2 at line 144
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 349
pkg/rpc/context.go in pkg/rpc.NewServerEx.func2.1 at line 145
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1504
pkg/rpc/auth.go in pkg/rpc.kvAuth.streamInterceptor at line 152
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1504
pkg/rpc/context.go in pkg/rpc.NewServerEx.func4 at line 187
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1504
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.StreamServerInterceptor.func13 at line 159
bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go in pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler at line 267
pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).RaftSnapshot at line 544
pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).raftSnapshot at line 566
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot at line 279
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 349
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot.func1 at line 282
pkg/kv/kvserver/store_snapshot.go in pkg/kv/kvserver.(*Store).receiveSnapshot at line 600
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest at line 499
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).withReplicaForRequest at line 435
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1 at line 558
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1155
pkg/kv/kvserver/replica_raftstorage.go in pkg/kv/kvserver.(*Replica).applySnapshotRaftMuLocked at line 758
pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).leasePostApplyLocked at line 362
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.3.4 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.3.4 |
| Cockroach SHA | 1cddb3a22695e967197844b214115eb645473efe |
| # of CPUs | 4 |
| # of Goroutines | 417 |

Jira issue: CRDB-57170