cockroachdb / cockroachdb/cockroach
importer: IMPORT INTO initial row count can fail with BatchTimestampBeforeGCError
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
Since 26.2, `IMPORT INTO` a non-empty table runs a full `SELECT count(*)` of the
target table before ingesting any data (`importResumer.detailsWithInitialRowCount`
in `pkg/sql/importer/import_job.go`). The count runs inside a `DescsTxn` with a
fixed read timestamp and no protected timestamp. If the scan takes longer than
the table's `gc.ttlseconds`, the MVCC GC threshold advances past the
transaction's read timestamp and the scan fails with
`BatchTimestampBeforeGCError`. The error is not retryable, so the whole IMPORT
job fails:
```
IMPORT job ...: stepping through state reverting with unexpected error: counting rows via synthetic descriptor: import-initial-row-count: batch timestamp 1789742889.361761000,0 must be after replica GC threshold 1789742889.362563000,0
```
Contributing factors:
- The count is unconditional. `bulkio.import.row_count_validation.mode = 'off'`
does not skip it, even though the resulting `InitialRowCount` is only consumed
by the post-import INSPECT row count check.
- The count is a distributed scan, one TableReader per node walking its local
ranges serially. Aggregate throughput scales with cluster size, but on a
table large enough it still runs for longer than a lowered GC TTL.
- Before 26.2 the same code path only did a one-row `Scan` to decide whether
the table was empty (used to pick ClearRange vs. DeleteRange on rollback), so
the cost was O(1) regardless of table size.
Every fresh IMPORT attempt repeats the count and fails the same way once the
table is large enough relative to its GC TTL, so affected users cannot import
into the table at all without raising `gc.ttlseconds`.
**To Reproduce**
1. Create a table, insert rows, and set `gc.ttlseconds` low enough that a full
scan of the table takes longer than the TTL.
2. `SET CLUSTER SETTING bulkio.import.row_count_validation.mode = 'off'`
(optional; it does not change the outcome).
3. `IMPORT INTO` the table.
4. The IMPORT fails with the error above once the GC threshold passes the count
transaction's read timestamp.
This can also be reproduced deterministically in a unit test without waiting on
the GC TTL: pause the count's first scan of the table with a store request
filter, advance the range's GC threshold with a `GCRequest`, then release the
scan.
**Expected behavior**
IMPORT INTO should not fail because its own pre-ingest read outlived the table's
GC TTL. Possible directions:
- When row count validation is off, skip the full count and fall back to the
one-row emptiness check.
- When validation is on, protect the count's read timestamp with a protected
timestamp record for the duration of the scan.
**Environment:**
- CockroachDB v26.2.x and later. Not present on 26.1.
**Additional context**
Related: #91151 tracks the general absence of a protected timestamp over IMPORT
spans, and #120897 / #110738 are the same error class from post-import
constraint validation.
Jira issue: CRDB-68506
Epic CRDB-68473
Contributor guide
Research direction
Start with importResumer.detailsWithInitialRowCount in pkg/sql/importer/import_job.go and trace how the DescsTxn performs the initial count. Review the existing TryToProtectBeforeGC path in INSPECT and the suggested deterministic test using a store request filter, GCRequest, and a paused scan. Done means IMPORT INTO no longer fails when this pre-ingest read outlives the GC TTL, with a regression test covering the failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 44/100