cockroachdb / cockroachdb/cockroach

Sentry: replica_closedts.go:295: log.Dev.Fatal: side-transport update saw closed timestamp regression on r429: (lai=256962, ts=1789607637.011648712,0) -> (lai=226809, ts=1789607675.837896536,0) (1) at...

Open
#175,568 1 comment 0 reactions 0 assignees View on GitHub
branch-release-25.4 C-bug O-sentry T-kv X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7736917552/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7736917552/?referrer=webhooks_plugin)

Panic Message:

```
replica_closedts.go:295: log.Dev.Fatal: side-transport update saw closed timestamp regression on r429: (lai=256962, ts=1789607637.011648712,0) -> (lai=226809, ts=1789607675.837896536,0)
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*sidetransportAccess).assertNoRegression
| pkg/kv/kvserver/replica_closedts.go:295
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*sidetransportAccess).forward
| pkg/kv/kvserver/replica_closedts.go:272
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).BumpSideTransportClosed
| pkg/kv/kvserver/replica_closedts.go:141
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport.(*Sender).publish
| pkg/kv/kvserver/closedts/sidetransport/sender.go:396
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport.(*Sender).Run.func2
| pkg/kv/kvserver/closedts/sidetransport/sender.go:259
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1
| pkg/util/stop/stopper.go:440
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (2) log.Dev.Fatal: side-transport update saw closed timestamp regression on r429: (lai=256962, ts=1789607637.011648712,0) -> (lai=226809, ts=1789607675.837896536,0)
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Dev.Fatal: side-transport update saw closed timestamp regression on r429: (lai=256962, ts=1789607637.011648712,0) -> (lai=226809, ts=1789607675.837896536,0)
replica_closedts.go:295: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[pkg/util/stop/stopper.go#L439-L441](pkg/util/stop/stopper.go#L439-L441)
[pkg/kv/kvserver/closedts/sidetransport/sender.go#L258-L260](pkg/kv/kvserver/closedts/sidetransport/sender.go#L258-L260)
[pkg/kv/kvserver/closedts/sidetransport/sender.go#L395-L397](pkg/kv/kvserver/closedts/sidetransport/sender.go#L395-L397)
[pkg/kv/kvserver/replica_closedts.go#L140-L142](pkg/kv/kvserver/replica_closedts.go#L140-L142)
[pkg/kv/kvserver/replica_closedts.go#L271-L273](pkg/kv/kvserver/replica_closedts.go#L271-L273)
[pkg/kv/kvserver/replica_closedts.go#L294-L296](pkg/kv/kvserver/replica_closedts.go#L294-L296)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1700
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1 at line 440
pkg/kv/kvserver/closedts/sidetransport/sender.go in pkg/kv/kvserver/closedts/sidetransport.(*Sender).Run.func2 at line 259
pkg/kv/kvserver/closedts/sidetransport/sender.go in pkg/kv/kvserver/closedts/sidetransport.(*Sender).publish at line 396
pkg/kv/kvserver/replica_closedts.go in pkg/kv/kvserver.(*Replica).BumpSideTransportClosed at line 141
pkg/kv/kvserver/replica_closedts.go in pkg/kv/kvserver.(*sidetransportAccess).forward at line 272
pkg/kv/kvserver/replica_closedts.go in pkg/kv/kvserver.(*sidetransportAccess).assertNoRegression at line 295
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.4.2 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.4.2 |
| Cockroach SHA | 2c80e22e93a1fc4e9c268fa7b6c2018acfa09724 |
| # of CPUs | 4 |
| # of Goroutines | 615 |

Jira issue: CRDB-68464

Contributor guide

Open the contributing guide

Research direction

Start by reading pkg/kv/kvserver/replica_closedts.go around assertNoRegression and forward, then trace the call from closedts/sidetransport/sender.go publish. Investigate how a lower lease application index reaches the side-transport update, and consider the reported Sentry failure on v25.4.2. Done means the regression is understood and a verified fix prevents this crash.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.