cockroachdb / cockroachdb/cockroach

changefeedccl: PTS management on Serverless

Open
#106,608 6 comments 0 reactions 0 assignees View on GitHub
A-cdc C-bug T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

As reported by serverless customer, changefeed failed with the replica GC error:

```
job failed (batch timestamp 1688843844.427894356,1 must be after replica GC threshold 1688992414.535792633,0) but is being paused because of on_error=pause
```

The changefeed seems to be well configured, and uses new webhook sink.
The job record does not have MaximumPTSAge set (more on this below).
Cluster is running 23.1.3 version. The changefeed utilizes CDC query (very simple one SELECT * WHERE status='hello')

The error could happen if something strange happening with protected timestamp record.
So, possibilities are:
* PTS management in changefeed is utterly broken (very unlikely)
* PTS management is broken on serverless (more likely)
* PTS management is broken when using CDC queries -- we do set txn timestamp when planning cdc query evaluation, and perhaps that's what's happening.

In addition, releases after 23.1.3 have changefeed.protect_timestamp.max_age which should expire
PTS record older than 4 days... but that functionality will certainly cause problems on serverless.

All of the above needs to be investigated/verified and fixed.

Jira issue: CRDB-29640

Epic CRDB-11783

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.