Sentry: replica_closedts.go:258: log.Fatal: side-transport update saw closed timestamp regression on r32138689: (lai=2309056, ts=1777999651.297847196,0) -> (lai=2309050, ts=1777999651.381654599,0) (1)...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7461189683/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7461189683/?referrer=webhooks_plugin)

Panic Message:

```
replica_closedts.go:258: log.Fatal: side-transport update saw closed timestamp regression on r32138689: (lai=2309056, ts=1777999651.297847196,0) -> (lai=2309050, ts=1777999651.381654599,0)
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*sidetransportAccess).assertNoRegression
| pkg/kv/kvserver/replica_closedts.go:258
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*sidetransportAccess).forward
| pkg/kv/kvserver/replica_closedts.go:235
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).ForwardSideTransportClosedTimestamp
| pkg/kv/kvserver/replica_closedts.go:133
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).ForwardSideTransportClosedTimestampForRange.func1
| pkg/kv/kvserver/stores.go:127
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores.func1
| pkg/kv/kvserver/stores.go:144
| github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Map[...]).Range
| pkg/util/syncutil/map.go:391
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores
| pkg/kv/kvserver/stores.go:143
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).ForwardSideTransportClosedTimestampForRange
| pkg/kv/kvserver/stores.go:124
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport.(*incomingStream).processUpdate
| pkg/kv/kvserver/closedts/sidetransport/receiver.go:249
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport.(*incomingStream).Run.func1
| pkg/kv/kvserver/closedts/sidetransport/receiver.go:327
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| pkg/util/stop/stopper.go:498
| runtime.goexit
| src/runtime/asm_arm64.s:1222
Wraps: (2) log.Fatal: side-transport update saw closed timestamp regression on r32138689: (lai=2309056, ts=1777999651.297847196,0) -> (lai=2309050, ts=1777999651.381654599,0)
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: side-transport update saw closed timestamp regression on r32138689: (lai=2309056, ts=1777999651.297847196,0) -> (lai=2309050, ts=1777999651.381654599,0)
replica_closedts.go:258: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_arm64.s#L1221-L1223](src/runtime/asm_arm64.s#L1221-L1223)
[pkg/util/stop/stopper.go#L497-L499](pkg/util/stop/stopper.go#L497-L499)
[pkg/kv/kvserver/closedts/sidetransport/receiver.go#L326-L328](pkg/kv/kvserver/closedts/sidetransport/receiver.go#L326-L328)
[pkg/kv/kvserver/closedts/sidetransport/receiver.go#L248-L250](pkg/kv/kvserver/closedts/sidetransport/receiver.go#L248-L250)
[pkg/kv/kvserver/stores.go#L123-L125](pkg/kv/kvserver/stores.go#L123-L125)
[pkg/kv/kvserver/stores.go#L142-L144](pkg/kv/kvserver/stores.go#L142-L144)
[pkg/util/syncutil/map.go#L390-L392](pkg/util/syncutil/map.go#L390-L392)
[pkg/kv/kvserver/stores.go#L143-L145](pkg/kv/kvserver/stores.go#L143-L145)
[pkg/kv/kvserver/stores.go#L126-L128](pkg/kv/kvserver/stores.go#L126-L128)
[pkg/kv/kvserver/replica_closedts.go#L132-L134](pkg/kv/kvserver/replica_closedts.go#L132-L134)
[pkg/kv/kvserver/replica_closedts.go#L234-L236](pkg/kv/kvserver/replica_closedts.go#L234-L236)
[pkg/kv/kvserver/replica_closedts.go#L257-L259](pkg/kv/kvserver/replica_closedts.go#L257-L259)

```
src/runtime/asm_arm64.s in runtime.goexit at line 1222
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 498
pkg/kv/kvserver/closedts/sidetransport/receiver.go in pkg/kv/kvserver/closedts/sidetransport.(*incomingStream).Run.func1 at line 327
pkg/kv/kvserver/closedts/sidetransport/receiver.go in pkg/kv/kvserver/closedts/sidetransport.(*incomingStream).processUpdate at line 249
pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).ForwardSideTransportClosedTimestampForRange at line 124
pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).VisitStores at line 143
pkg/util/syncutil/map.go in pkg/util/syncutil.(*Map[...]).Range at line 391
pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).VisitStores.func1 at line 144
pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).ForwardSideTransportClosedTimestampForRange.func1 at line 127
pkg/kv/kvserver/replica_closedts.go in pkg/kv/kvserver.(*Replica).ForwardSideTransportClosedTimestamp at line 133
pkg/kv/kvserver/replica_closedts.go in pkg/kv/kvserver.(*sidetransportAccess).forward at line 235
pkg/kv/kvserver/replica_closedts.go in pkg/kv/kvserver.(*sidetransportAccess).assertNoRegression at line 258
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v24.3.18 |
| Go Version | go1.22.8 X:nocoverageredesign |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v24.3.18 |
| Cockroach SHA | 37439c410d4583708c898b13a2f9e906f444867b |
| # of CPUs | 48 |
| # of Goroutines | 9400 |

Jira issue: CRDB-63640