cockroachdb / cockroachdb/cockroach
Changefeed failures seen on drt-chaos internal CRDB cluster
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
We have seen test changefeed failures in our internal cluster drt-chaos. The failure is
```
HANGEFEED job 979215929488343050: stepping through state reverting with unexpected error: received boundary timestamp 1721711057.579153634,0 < 1721716400.060881844,0 of type ‹BACKFILL› before reaching existing boundary of type ‹RESTART›
(1) assertion failure
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*schemaChangeFrontier).ForwardResolvedSpan
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:1932
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeFrontier).forwardFrontier
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:1542
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeFrontier).noteAggregatorProgress
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:1533
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeFrontier).Next
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:1494
| github.com/cockroachdb/cockroach/pkg/sql/execinfra.Run
| github.com/cockroachdb/cockroach/pkg/sql/execinfra/base.go:197
| github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run
| github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:732
| github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run
| github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:579
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:928
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.startDistChangefeed.func1
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_dist.go:328
| github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1
| github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168
| golang.org/x/sync/errgroup.(*Group).Go.func1
| golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (3) received boundary timestamp 1721711057.579153634,0 < 1721716400.060881844,0 of type ‹BACKFILL› before reaching existing boundary of type ‹RESTART›
Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
HINT: ‹You have encountered an unexpected error.›
```
More details can be found in the slack thread - https://cockroachlabs.slack.com/archives/C05FHJJ0MD0/p1721805027341599
Jira issue: CRDB-40631
Epic CRDB-41785
Contributor guide
Assessment
This issue has not been assessed yet.