cockroachdb / cockroachdb/cockroach
logical,txnmode: report catchup,lagging ranges to job progress
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Similar to how we report initial/catchup scans to the DBConsole/job progress, we should do the same for transactional mode LDR.
The main complexity in implementing this is deciding how to propagate the rangestats from the coordinator/mergefeed (where the rangefeeds emit to) to the checkpoint handler (where we update the job). Unlike classic LDR where each writer gets rangefeed stats and we naturally fan into the checkpoint handler, transactional LDR starts with all the rangefeed stats already aggregated, and we have to fan out to multiple appliers before fanning back into the checkpoint handler.
The main solution I considered so far is leveraging DistSQL metadata. We could create a `oneof` "progress" protobuf and teach the checkpoint handler to handle multiple types of progress updates. We already represent applier frontier updates as metadata; rangestats updates would be another. This would let us plumb coordinator specific information through the applier without exposing it directly like piggybacking off `LDRApplierEvents` would.
Note that as of right now, transactional mode LDR doesn't actually support initial scans. Seeing initial scan progress is the main motivation for displaying this information so we might only consider implementing this once we do support initial scans.
Jira issue: CRDB-64879
Epic CRDB-65552
Contributor guide
Assessment
This issue has not been assessed yet.