cockroachdb / cockroachdb/cockroach
Sentry: replica_closedts.go:295: log.Dev.Fatal: side-transport update saw closed timestamp regression on r429: (lai=256962, ts=1789607637.011648712,0) -> (lai=226809, ts=1789607675.837896536,0) (1) at...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/7736917552/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7736917552/?referrer=webhooks_plugin)
Panic Message:
```
replica_closedts.go:295: log.Dev.Fatal: side-transport update saw closed timestamp regression on r429: (lai=256962, ts=1789607637.011648712,0) -> (lai=226809, ts=1789607675.837896536,0)
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*sidetransportAccess).assertNoRegression
| pkg/kv/kvserver/replica_closedts.go:295
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*sidetransportAccess).forward
| pkg/kv/kvserver/replica_closedts.go:272
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).BumpSideTransportClosed
| pkg/kv/kvserver/replica_closedts.go:141
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport.(*Sender).publish
| pkg/kv/kvserver/closedts/sidetransport/sender.go:396
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport.(*Sender).Run.func2
| pkg/kv/kvserver/closedts/sidetransport/sender.go:259
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1
| pkg/util/stop/stopper.go:440
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (2) log.Dev.Fatal: side-transport update saw closed timestamp regression on r429: (lai=256962, ts=1789607637.011648712,0) -> (lai=226809, ts=1789607675.837896536,0)
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Dev.Fatal: side-transport update saw closed timestamp regression on r429: (lai=256962, ts=1789607637.011648712,0) -> (lai=226809, ts=1789607675.837896536,0)
replica_closedts.go:295: *withstack.withStack (top exception)
```
Stacktrace (expand for inline code snippets):
[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[pkg/util/stop/stopper.go#L439-L441](pkg/util/stop/stopper.go#L439-L441)
[pkg/kv/kvserver/closedts/sidetransport/sender.go#L258-L260](pkg/kv/kvserver/closedts/sidetransport/sender.go#L258-L260)
[pkg/kv/kvserver/closedts/sidetransport/sender.go#L395-L397](pkg/kv/kvserver/closedts/sidetransport/sender.go#L395-L397)
[pkg/kv/kvserver/replica_closedts.go#L140-L142](pkg/kv/kvserver/replica_closedts.go#L140-L142)
[pkg/kv/kvserver/replica_closedts.go#L271-L273](pkg/kv/kvserver/replica_closedts.go#L271-L273)
[pkg/kv/kvserver/replica_closedts.go#L294-L296](pkg/kv/kvserver/replica_closedts.go#L294-L296)
```
src/runtime/asm_amd64.s in runtime.goexit at line 1700
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1 at line 440
pkg/kv/kvserver/closedts/sidetransport/sender.go in pkg/kv/kvserver/closedts/sidetransport.(*Sender).Run.func2 at line 259
pkg/kv/kvserver/closedts/sidetransport/sender.go in pkg/kv/kvserver/closedts/sidetransport.(*Sender).publish at line 396
pkg/kv/kvserver/replica_closedts.go in pkg/kv/kvserver.(*Replica).BumpSideTransportClosed at line 141
pkg/kv/kvserver/replica_closedts.go in pkg/kv/kvserver.(*sidetransportAccess).forward at line 272
pkg/kv/kvserver/replica_closedts.go in pkg/kv/kvserver.(*sidetransportAccess).assertNoRegression at line 295
```
### Tags
| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.4.2 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.4.2 |
| Cockroach SHA | 2c80e22e93a1fc4e9c268fa7b6c2018acfa09724 |
| # of CPUs | 4 |
| # of Goroutines | 615 |
Jira issue: CRDB-68464
Contributor guide
Research direction
Start by reading pkg/kv/kvserver/replica_closedts.go around assertNoRegression and forward, then trace the call from closedts/sidetransport/sender.go publish. Investigate how a lower lease application index reaches the side-transport update, and consider the reported Sentry failure on v25.4.2. Done means the regression is understood and a verified fix prevents this crash.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100