cockroachdb / cockroachdb/cockroach
sql: log periodic progress during IMPORT revert
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
## Motivation
When an `IMPORT INTO` on a non-empty table is canceled or fails, revert runs through `DeleteTableWithPredicate` in `pkg/sql/revert.go`. For large tables this can take hours. Today there is no way to tell whether revert is progressing, stuck, or near completion without:
- Polling `SHOW RANGES ... WITH DETAILS` and watching `live_count` decline (requires manual queries, table must be inspectable).
- Pulling a stacks dump and inspecting goroutines in `DeleteTableWithPredicate`.
- Cranking `vmodule=revert=2` mid-flight to surface per-range `VEventf` lines, which is far too verbose for large reverts.
Operators investigating slow/wedged reverts have no built-in signal of forward progress. Even modest periodic log lines would be a major step up.
## Suggested approach
The function already has all the data it needs in-process. A minimal implementation:
1. **Count total ranges up front.** The producer already does a `RangeIterator` pass to dispatch spans. Add a quick separate pass (or count during the dispatch pass and log totals once known) to get a denominator. This is cheap compared to the actual revert work.
2. **Track counters via atomics:**
- `batchesDispatched` — incremented by the producer.
- `batchesCompleted` — incremented by workers after each successful `DeleteRange` RPC (or after a batch with no resume span).
- `resumeSpansSeen` — useful signal that actual work-per-batch is exceeding the 500K-key cap and the revert will take longer than the batch count suggests.
- `keysDeleted` (optional) — sum of `NumKeys` from each `DeleteRangeResponse`.
3. **Spawn a logging ticker** alongside the worker group, e.g. every 30s or 1m:
```
I revert.go:NN deleting table T: dispatched B/Btotal batches, completed C, resume spans seen R, ~K keys deleted, elapsed Xs
```
4. **Log on entry and exit** with the predicate, total range count, settings (`predicateDeleteRangeNumWorkers`, `rollbackBatchSize`), so operators can correlate progress to tunables without reading code.
## Optional extension — surface to job progress
`DeleteTableWithPredicate` is invoked from `revertTable` inside the IMPORT job's `OnFailOrCancel`. Plumbing the job through (or a `progressFn func(fraction float32)` callback) would let revert call `job.FractionProgressed(...)`, making progress visible in `SHOW JOBS` and the DB Console. This is more invasive — the function currently has no job dependency — but it is the natural follow-on.
## Non-goals
- Full metrics/UI. This issue is just about emitting enough information to a log to answer "is revert making progress, and roughly how much is left?" during a live incident.
## Caveats worth noting in the implementation
- Per-batch work is variable — a single batch can produce multiple `DeleteRange` RPCs (one per resume span) capped at `RevertTableDefaultBatchSize` (500K keys) each. `batchesCompleted`/`batchesDispatched` is a coarse fraction; `resumeSpansSeen` exposes the variance.
- The total range count computed up front can drift if splits/merges happen during revert. Treat it as a best-effort denominator, not a hard invariant.
Jira issue: CRDB-63116
Epic CRDB-65516
Contributor guide
Assessment
This issue has not been assessed yet.