cockroachdb / cockroachdb/cockroach

importer: job restart resets checkpointed resume positions, so a second interruption re-imports completed files

Open
#174,963 1 comment 0 reactions 0 assignees View on GitHub
A-import C-bug O-agent O-roachtest T-sql-foundations
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

### NOTE: This is identified and filed by claude, so take it with a grain of salt

**Describe the problem**

When an IMPORT job is restarted, whether by PAUSE/RESUME, a coordinator restart,
or re-adoption after the coordinator's SQL liveness session expires, the first
progress checkpoint of the new attempt overwrites the job's persisted per-file
resume positions with zeros. The running attempt is unaffected, since its
processors were planned from the record as it stood before the overwrite, but
the persisted checkpoint now claims that files completed in earlier attempts
were never started. A second interruption therefore plans from that record and
imports those files again from offset 0, and when interruptions recur faster
than a full pass the job never converges.

This is the same defect class as #173109, which fixed the declarative schema
changer's merge checkpoint: the in-memory accumulator is seeded empty instead of
from the durable checkpoint, and the durable checkpoint is replaced rather than
merged.

The mechanism, as of master @ 8812064a (identical on release-26.3), has three
parts:

1. Planning seeds each processor's `ResumePos` from the job record
([import_processor_planning.go#L706](https://github.com/cockroachdb/cockroach/blob/8812064a015d2faf99d3fc7e15880f94042954b0/pkg/sql/importer/import_processor_planning.go#L706))
and keeps files whose position is `math.MaxInt64` in the spec. The reader
skips those files
([read_import_base.go#L107-L110](https://github.com/cockroachdb/cockroach/blob/8812064a015d2faf99d3fc7e15880f94042954b0/pkg/sql/importer/read_import_base.go#L107-L110))
and marks them complete again only once every file of the processor is done
([read_import_base.go#L140](https://github.com/cockroachdb/cockroach/blob/8812064a015d2faf99d3fc7e15880f94042954b0/pkg/sql/importer/read_import_base.go#L140)).
2. The coordinator's `importCheckpointTracker` is constructed with zeroed
`rowProgress` and `fractionProgress`
([import_progress_tracker.go#L58-L66](https://github.com/cockroachdb/cockroach/blob/8812064a015d2faf99d3fc7e15880f94042954b0/pkg/sql/importer/import_progress_tracker.go#L58-L66))
and is never seeded from the persisted `ResumePos`/`ReadProgress`.
`RecordProcessorUpdate` overwrites each reported file unconditionally
([#L95](https://github.com/cockroachdb/cockroach/blob/8812064a015d2faf99d3fc7e15880f94042954b0/pkg/sql/importer/import_progress_tracker.go#L95)),
and `Persist` copies the tracker wholesale over `prog.ResumePos` and
`prog.ReadProgress` and recomputes `FractionCompleted`
([#L199-L201](https://github.com/cockroachdb/cockroach/blob/8812064a015d2faf99d3fc7e15880f94042954b0/pkg/sql/importer/import_progress_tracker.go#L199-L201)).
The progress ticker calls `Persist` 10 s after the attempt starts
([import_processor_planning.go#L354-L364](https://github.com/cockroachdb/cockroach/blob/8812064a015d2faf99d3fc7e15880f94042954b0/pkg/sql/importer/import_processor_planning.go#L354-L364)).
3. Each processor's `importProgressTracker` likewise starts every file in
`spec.Uri` at zero without consulting `spec.ResumePos`
([import_processor.go#L599](https://github.com/cockroachdb/cockroach/blob/8812064a015d2faf99d3fc7e15880f94042954b0/pkg/sql/importer/import_processor.go#L599),
[#L610](https://github.com/cockroachdb/cockroach/blob/8812064a015d2faf99d3fc7e15880f94042954b0/pkg/sql/importer/import_processor.go#L610)),
and `formatProgress` reports every file in the spec
([#L633](https://github.com/cockroachdb/cockroach/blob/8812064a015d2faf99d3fc7e15880f94042954b0/pkg/sql/importer/import_processor.go#L633)),
so files it never reads, which are exactly the completed ones, are reported
as position 0 and fraction 0 on every push. This is what makes the reset
durable for the rest of the attempt rather than a 10 s blip.

**To Reproduce**

Derived from the code and the roachtest run below; not yet reduced to a unit
test.

1. IMPORT a table from several files, with more than one file per processor.
2. After at least one file has completed, interrupt the job (PAUSE JOB then
RESUME JOB, or restart the coordinator) and let the new attempt run for more
than 10 s.
3. Observe that the `SHOW JOB` fraction drops to roughly 0 and that
`ImportProgress.ResumePos` for the completed files reads 0 instead of
`math.MaxInt64`.
4. Interrupt again and observe the new attempt re-reading the completed files
from offset 0.

`TestCSVImportCanBeResumed` and `TestImportHonorsResumePosition` each interrupt
exactly once, so neither catches the regression; extending one of them to a
second interruption with an assertion on `ResumePos` should fail today.

**Expected behavior**

The persisted checkpoint is monotonic across attempts: a completed file stays
marked complete, an in-progress file never regresses below its last flushed
position, and a restart re-reads at most the unflushed tail of in-progress
files.

**Additional data / screenshots**

Observed in roachtest `import/pause/distmerge=false/nodes=4` (#174633, run of
2026-09-08) importing tpch/lineitem on 4 nodes with 4 vCPUs each, where cluster
overload made the coordinator's SQL liveness session expire repeatedly and the
jobs registry re-adopted the job seven times in 64 minutes.
`system.job_progress_history` for the job:

| time (UTC) | fraction | event |
|---|---|---|
| 09:04:04 | 0.323 | last checkpoint of attempt 1 |
| 09:04:04 | | coordinator re-adopts the job under a new session |
| 09:04:16 | 0.0008 | first ticker `Persist` of attempt 2 |
| 09:06:39 | 0.164 | attempt 2's processors report in-progress files; completed files stay at 0 |
| 09:07:01 | | another node adopts the job (attempt 3) |
| 09:07:11 | 0.0008 | first `Persist` of attempt 3, planned from the reset record |
| 10:13:51 | 0.0025 | after five more re-adoptions |

Each of the four PAUSE/RESUME cycles earlier in the same run also dropped the
fraction to 0 ten seconds after the resume (07:35:00, 07:36:01, 07:36:47), which
is the single-interruption, cosmetic form of the same bug.

**Environment:**
- CockroachDB v26.3 development build; the code on master is unchanged. The
defect predates c5fc818bd33 (2026-03-11, "importer: extract
importCheckpointTracker from distImport"), whose parent zero-initialized the
same slice and copied it over `ResumePos` the same way.

**Additional context**

Correctness is not affected, as IMPORT writes at a fixed timestamp and
re-ingesting a file produces identical KVs; the cost is wall-clock time, and
non-convergence when interruptions are frequent. A likely secondary effect,
not verified, is inflated row counts in the job summary, since planning only
resets the accumulated `BulkOpSummary` when every file reads position 0.

A fix would seed `importCheckpointTracker` from the persisted `ResumePos` and
`ReadProgress` at construction; seed the processor's `importProgressTracker`
from `spec.ResumePos`, reporting position `math.MaxInt64` and fraction 1.0 for
completed files or omitting unread files from `formatProgress`; and make
`RecordProcessorUpdate` and `Persist` never lower a position, as a backstop
against reordered reports.

Jira issue: CRDB-68134

Epic CRDB-65516

Contributor guide

Open the contributing guide

Research direction

Start with pkg/sql/importer/import_progress_tracker.go and the construction and reporting paths in import_processor_planning.go, import_processor.go, and read_import_base.go. Read TestCSVImportCanBeResumed and TestImportHonorsResumePosition first, then extend coverage to a second interruption. Done means persisted ResumePos and ReadProgress never regress across attempts and completed files are not re-imported from offset 0.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.