cockroachdb / cockroachdb/cockroach
changefeedccl: PTS management on Serverless
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
As reported by serverless customer, changefeed failed with the replica GC error:
```
job failed (batch timestamp 1688843844.427894356,1 must be after replica GC threshold 1688992414.535792633,0) but is being paused because of on_error=pause
```
The changefeed seems to be well configured, and uses new webhook sink.
The job record does not have MaximumPTSAge set (more on this below).
Cluster is running 23.1.3 version. The changefeed utilizes CDC query (very simple one SELECT * WHERE status='hello')
The error could happen if something strange happening with protected timestamp record.
So, possibilities are:
* PTS management in changefeed is utterly broken (very unlikely)
* PTS management is broken on serverless (more likely)
* PTS management is broken when using CDC queries -- we do set txn timestamp when planning cdc query evaluation, and perhaps that's what's happening.
In addition, releases after 23.1.3 have changefeed.protect_timestamp.max_age which should expire
PTS record older than 4 days... but that functionality will certainly cause problems on serverless.
All of the above needs to be investigated/verified and fixed.
Jira issue: CRDB-29640
Epic CRDB-11783
Contributor guide
Assessment
This issue has not been assessed yet.