cockroachdb / cockroachdb/cockroach

Changefeed failures seen on drt-chaos internal CRDB cluster

Open
#127,839 3 comments 0 reactions 0 assignees View on GitHub
A-cdc branch-release-24.2 C-bug O-testcluster T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We have seen test changefeed failures in our internal cluster drt-chaos. The failure is
```
HANGEFEED job 979215929488343050: stepping through state reverting with unexpected error: received boundary timestamp 1721711057.579153634,0 < 1721716400.060881844,0 of type ‹BACKFILL› before reaching existing boundary of type ‹RESTART›
(1) assertion failure
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*schemaChangeFrontier).ForwardResolvedSpan
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:1932
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeFrontier).forwardFrontier
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:1542
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeFrontier).noteAggregatorProgress
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:1533
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeFrontier).Next
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:1494
| github.com/cockroachdb/cockroach/pkg/sql/execinfra.Run
| github.com/cockroachdb/cockroach/pkg/sql/execinfra/base.go:197
| github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run
| github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:732
| github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run
| github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:579
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:928
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.startDistChangefeed.func1
| github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_dist.go:328
| github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1
| github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168
| golang.org/x/sync/errgroup.(*Group).Go.func1
| golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (3) received boundary timestamp 1721711057.579153634,0 < 1721716400.060881844,0 of type ‹BACKFILL› before reaching existing boundary of type ‹RESTART›
Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
HINT: ‹You have encountered an unexpected error.›
```

More details can be found in the slack thread - https://cockroachlabs.slack.com/archives/C05FHJJ0MD0/p1721805027341599

Jira issue: CRDB-40631

Epic CRDB-41785

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.