Sentry: replica_closedts.go:263: log.Fatal: side-transport update saw closed timestamp regression on r3: (lai=1855446, ts=1773120488.294536640,0) -> (lai=1817090, ts=1773120518.215913947,0) (1) attach...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7323513248/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7323513248/?referrer=webhooks_plugin)

Panic Message:

```
replica_closedts.go:263: log.Fatal: side-transport update saw closed timestamp regression on r3: (lai=1855446, ts=1773120488.294536640,0) -> (lai=1817090, ts=1773120518.215913947,0)
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*sidetransportAccess).assertNoRegression
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go:263
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*sidetransportAccess).forward
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go:237
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*sidetransportAccess).get
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go:334
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).getCurrentClosedTimestampLocked
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_follower_read.go:144
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).GetCurrentClosedTimestamp
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_follower_read.go:157
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).updateReplicationGauges.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:3315
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*storeReplicaVisitor).Visit
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:563
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).updateReplicationGauges
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:3238
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).computeMetrics
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:3478
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).ComputeMetricsPeriodically
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:3519
| github.com/cockroachdb/cockroach/pkg/server.(*Node).computeMetricsPeriodically.func1
| github.com/cockroachdb/cockroach/pkg/server/node.go:1035
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:150
| github.com/cockroachdb/cockroach/pkg/util/syncutil.(*IntMap).Range
| github.com/cockroachdb/cockroach/pkg/util/syncutil/int_map.go:385
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:149
| github.com/cockroachdb/cockroach/pkg/server.(*Node).computeMetricsPeriodically
| github.com/cockroachdb/cockroach/pkg/server/node.go:1034
| github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func2
| github.com/cockroachdb/cockroach/pkg/server/node.go:1007
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (2) log.Fatal: side-transport update saw closed timestamp regression on r3: (lai=1855446, ts=1773120488.294536640,0) -> (lai=1817090, ts=1773120518.215913947,0)
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: side-transport update saw closed timestamp regression on r3: (lai=1855446, ts=1773120488.294536640,0) -> (lai=1817090, ts=1773120518.215913947,0)
replica_closedts.go:263: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/util/stop/stopper.go#L484-L486](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/util/stop/stopper.go#L484-L486)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/server/node.go#L1006-L1008](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/server/node.go#L1006-L1008)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/server/node.go#L1033-L1035](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/server/node.go#L1033-L1035)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L148-L150](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L148-L150)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/util/syncutil/int_map.go#L384-L386](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/util/syncutil/int_map.go#L384-L386)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L149-L151](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L149-L151)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/server/node.go#L1034-L1036](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/server/node.go#L1034-L1036)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store.go#L3518-L3520](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store.go#L3518-L3520)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store.go#L3477-L3479](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store.go#L3477-L3479)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store.go#L3237-L3239](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store.go#L3237-L3239)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store.go#L562-L564](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store.go#L562-L564)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store.go#L3314-L3316](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/store.go#L3314-L3316)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_follower_read.go#L156-L158](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_follower_read.go#L156-L158)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_follower_read.go#L143-L145](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_follower_read.go#L143-L145)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go#L333-L335](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go#L333-L335)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go#L236-L238](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go#L236-L238)
[https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go#L262-L264](https://github.com/cockroachdb/cockroach/blob/9e10212477fde97f55ea4fff01797288c836575c/pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go#L262-L264)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 485
pkg/server/node.go in pkg/server.(*Node).startComputePeriodicMetrics.func2 at line 1007
pkg/server/node.go in pkg/server.(*Node).computeMetricsPeriodically at line 1034
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).VisitStores at line 149
pkg/util/syncutil/int_map.go in pkg/util/syncutil.(*IntMap).Range at line 385
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).VisitStores.func1 at line 150
pkg/server/node.go in pkg/server.(*Node).computeMetricsPeriodically.func1 at line 1035
pkg/kv/kvserver/pkg/kv/kvserver/store.go in pkg/kv/kvserver.(*Store).ComputeMetricsPeriodically at line 3519
pkg/kv/kvserver/pkg/kv/kvserver/store.go in pkg/kv/kvserver.(*Store).computeMetrics at line 3478
pkg/kv/kvserver/pkg/kv/kvserver/store.go in pkg/kv/kvserver.(*Store).updateReplicationGauges at line 3238
pkg/kv/kvserver/pkg/kv/kvserver/store.go in pkg/kv/kvserver.(*storeReplicaVisitor).Visit at line 563
pkg/kv/kvserver/pkg/kv/kvserver/store.go in pkg/kv/kvserver.(*Store).updateReplicationGauges.func1 at line 3315
pkg/kv/kvserver/pkg/kv/kvserver/replica_follower_read.go in pkg/kv/kvserver.(*Replica).GetCurrentClosedTimestamp at line 157
pkg/kv/kvserver/pkg/kv/kvserver/replica_follower_read.go in pkg/kv/kvserver.(*Replica).getCurrentClosedTimestampLocked at line 144
pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go in pkg/kv/kvserver.(*sidetransportAccess).get at line 334
pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go in pkg/kv/kvserver.(*sidetransportAccess).forward at line 237
pkg/kv/kvserver/pkg/kv/kvserver/replica_closedts.go in pkg/kv/kvserver.(*sidetransportAccess).assertNoRegression at line 263
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v24.1.3 |
| Go Version | go1.22.5 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v24.1.3 |
| Cockroach SHA | 9e10212477fde97f55ea4fff01797288c836575c |
| # of CPUs | 12 |
| # of Goroutines | 560 |

Jira issue: CRDB-61302

Start with pkg/kv/kvserver/replica_closedts.go at sidetransportAccess.assertNoRegression, then trace the reported call path through replica_follower_read.go and store.go. Determine why the closed timestamp regressed on the live cluster, reproduce or characterize it, and ensure the fatal error is resolved with appropriate regression coverage.

Written by the indexing model from the issue text.