Sentry: replica_closedts.go:263: log.Fatal: side-transport update saw closed timestamp regression on r140: (lai=1018814, ts=1772722563.933794903,0) -> (lai=1018803, ts=1772722689.929755809,0) (1) atta...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7312613831/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7312613831/?referrer=webhooks_plugin)

Panic Message:

```
replica_closedts.go:263: log.Fatal: side-transport update saw closed timestamp regression on r140: (lai=1018814, ts=1772722563.933794903,0) -> (lai=1018803, ts=1772722689.929755809,0)
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*sidetransportAccess).assertNoRegression
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go:263
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*sidetransportAccess).forward
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go:237
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).ForwardSideTransportClosedTimestamp
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go:138
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).ForwardSideTransportClosedTimestampForRange.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:133
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:150
| github.com/cockroachdb/cockroach/pkg/util/syncutil.(*IntMap).Range
| github.com/cockroachdb/cockroach/pkg/util/syncutil/int_map.go:385
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:149
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).ForwardSideTransportClosedTimestampForRange
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:130
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport.(*incomingStream).processUpdate
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport/receiver.go:254
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport.(*incomingStream).Run.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport/receiver.go:332
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (2) log.Fatal: side-transport update saw closed timestamp regression on r140: (lai=1018814, ts=1772722563.933794903,0) -> (lai=1018803, ts=1772722689.929755809,0)
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: side-transport update saw closed timestamp regression on r140: (lai=1018814, ts=1772722563.933794903,0) -> (lai=1018803, ts=1772722689.929755809,0)
replica_closedts.go:263: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/util/stop/stopper.go#L484-L486](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/util/stop/stopper.go#L484-L486)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/closedts/sidetransport/receiver.go#L331-L333](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/closedts/sidetransport/receiver.go#L331-L333)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/closedts/sidetransport/receiver.go#L253-L255](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/closedts/sidetransport/receiver.go#L253-L255)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L129-L131](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L129-L131)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L148-L150](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L148-L150)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/util/syncutil/int_map.go#L384-L386](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/util/syncutil/int_map.go#L384-L386)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L149-L151](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L149-L151)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L132-L134](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L132-L134)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go#L137-L139](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go#L137-L139)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go#L236-L238](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go#L236-L238)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go#L262-L264](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go#L262-L264)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 485
pkg/kv/kvserver/closedts/sidetransport/receiver.go in pkg/kv/kvserver/closedts/sidetransport.(*incomingStream).Run.func1 at line 332
pkg/kv/kvserver/closedts/sidetransport/receiver.go in pkg/kv/kvserver/closedts/sidetransport.(*incomingStream).processUpdate at line 254
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).ForwardSideTransportClosedTimestampForRange at line 130
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).VisitStores at line 149
pkg/util/syncutil/int_map.go in pkg/util/syncutil.(*IntMap).Range at line 385
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).VisitStores.func1 at line 150
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).ForwardSideTransportClosedTimestampForRange.func1 at line 133
pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go in pkg/kv/kvserver.(*Replica).ForwardSideTransportClosedTimestamp at line 138
pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go in pkg/kv/kvserver.(*sidetransportAccess).forward at line 237
pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go in pkg/kv/kvserver.(*sidetransportAccess).assertNoRegression at line 263
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v24.1.3 |
| Go Version | go1.22.5 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v24.1.3 |
| Cockroach SHA | 9e10212477fde97f55ea4fff01797288c836575c |
| # of CPUs | 12 |
| # of Goroutines | 675 |

Jira issue: CRDB-61082

Start in pkg/kv/kvserver/replica_closedts.go at sidetransportAccess.assertNoRegression and trace the update through forward and ForwardSideTransportClosedTimestamp. Then follow receiver.go and stores.go to understand how the regression reaches the replica. Done means the reported closed-timestamp regression is explained and fixed, with coverage demonstrating that side-transport updates do not regress.

Written by the indexing model from the issue text.