cockroachdb / cockroachdb/cockroach

sql: log periodic progress during IMPORT revert

Open
#168,757 0 comments 0 reactions 0 assignees View on GitHub
C-enhancement T-sql-foundations
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

## Motivation

When an `IMPORT INTO` on a non-empty table is canceled or fails, revert runs through `DeleteTableWithPredicate` in `pkg/sql/revert.go`. For large tables this can take hours. Today there is no way to tell whether revert is progressing, stuck, or near completion without:

- Polling `SHOW RANGES ... WITH DETAILS` and watching `live_count` decline (requires manual queries, table must be inspectable).
- Pulling a stacks dump and inspecting goroutines in `DeleteTableWithPredicate`.
- Cranking `vmodule=revert=2` mid-flight to surface per-range `VEventf` lines, which is far too verbose for large reverts.

Operators investigating slow/wedged reverts have no built-in signal of forward progress. Even modest periodic log lines would be a major step up.

## Suggested approach

The function already has all the data it needs in-process. A minimal implementation:

1. **Count total ranges up front.** The producer already does a `RangeIterator` pass to dispatch spans. Add a quick separate pass (or count during the dispatch pass and log totals once known) to get a denominator. This is cheap compared to the actual revert work.
2. **Track counters via atomics:**
- `batchesDispatched` — incremented by the producer.
- `batchesCompleted` — incremented by workers after each successful `DeleteRange` RPC (or after a batch with no resume span).
- `resumeSpansSeen` — useful signal that actual work-per-batch is exceeding the 500K-key cap and the revert will take longer than the batch count suggests.
- `keysDeleted` (optional) — sum of `NumKeys` from each `DeleteRangeResponse`.
3. **Spawn a logging ticker** alongside the worker group, e.g. every 30s or 1m:
```
I revert.go:NN deleting table T: dispatched B/Btotal batches, completed C, resume spans seen R, ~K keys deleted, elapsed Xs
```
4. **Log on entry and exit** with the predicate, total range count, settings (`predicateDeleteRangeNumWorkers`, `rollbackBatchSize`), so operators can correlate progress to tunables without reading code.

## Optional extension — surface to job progress

`DeleteTableWithPredicate` is invoked from `revertTable` inside the IMPORT job's `OnFailOrCancel`. Plumbing the job through (or a `progressFn func(fraction float32)` callback) would let revert call `job.FractionProgressed(...)`, making progress visible in `SHOW JOBS` and the DB Console. This is more invasive — the function currently has no job dependency — but it is the natural follow-on.

## Non-goals

- Full metrics/UI. This issue is just about emitting enough information to a log to answer "is revert making progress, and roughly how much is left?" during a live incident.

## Caveats worth noting in the implementation

- Per-batch work is variable — a single batch can produce multiple `DeleteRange` RPCs (one per resume span) capped at `RevertTableDefaultBatchSize` (500K keys) each. `batchesCompleted`/`batchesDispatched` is a coarse fraction; `resumeSpansSeen` exposes the variance.
- The total range count computed up front can drift if splits/merges happen during revert. Treat it as a best-effort denominator, not a hard invariant.

Jira issue: CRDB-63116

Epic CRDB-65516

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.