cockroachdb / cockroachdb/cockroach
streamingccl: batch timestamp error not returned to client
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
In a recent replication stream failure, the job went into a paused state with an error that looked like:
```
pausing due to error; use RESUME JOB to try to proceed once the issue is resolved, or CANCEL JOB to rollback: timeout: context canceled
```
But, when digging into the logs, it appears that the real error was:
```
logs/cockroach.cct-232-dest-0004.ubuntu.2023-11-02T21_18_39Z.012485.log:E231103 03:49:26.987900 2425145 ccl/streamingccl/streamingest/stream_ingestion_processor.go:556 ⋮ [T1,Vsystem,n4,f‹a35bb8dd›,job=913868579942367233,distsql.gateway=9,distsql.appname=‹$ internal-resume-job-913868579942367233›] 41872 error on close(): ‹ERROR: batch timestamp 1698960957.937065386,0 must be after replica GC threshold 1698967771.741796894,0 (SQLSTATE XXUUU)›
```
This error should have been raised to the overall job. Further, we likely should make this a permanent error that we don't retry since no amount of retry is going to fix it.
Jira issue: CRDB-33238
Contributor guide
Assessment
This issue has not been assessed yet.