pingcap / pingcap/tidb

[import] Global Sort IMPORT can omit a KV group after runtime downscale and leave ghost unique indexes

Open
#70,576 1 comment 0 reactions 1 assignee Claimed by @D3Hunter View on GitHub
component/import found-by-ai may-affects-25.10 may-affects-26.3 may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/moderate type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

# [import] Global Sort IMPORT can omit a KV group after runtime downscale and leave ghost unique indexes

## Bug Report

### 1. Minimal reproduce step (Required)

This issue needs three ordinary ingredients:

1. Global Sort `IMPORT INTO` without forced merge.
2. A completed merge plan where one KV group skips merge at 4 runtime slots, but would require merge at 1 slot.
3. A supported `max_runtime_slots` downscale during a non-trivial merge, followed by a TiDB restart or task retry before write-and-ingest planning reloads task metadata.

The production-shaped sequence is:

1. Start a large Global Sort import.
2. Let merge planning run while `GetRuntimeSlots()` is 4, so one KV group is intentionally left in the encode output and another KV group still goes through a real merge subtask.
3. During that merge stage, use the documented throttle:

```text
POST /dxf/task/{taskID}/max_runtime_slots?value=1
```

4. Before write-and-ingest planning, let TiDB restart or let the task retry so the next planning context reloads `MaxRuntimeSlots=1`.

After the reload, write-and-ingest re-evaluates the old skip/merge predicate under the new slot count instead of consuming the completed merge output map.

For maintainers, the current-master behavior was made deterministic with regression probes around:

- `pkg/dxf/importinto/planner.go`
- `pkg/dxf/importinto/scheduler.go`
- `pkg/dxf/importinto/planner_test.go`
- `pkg/dxf/importinto/scheduler_testkit_test.go`
- `tests/realtikvtest/importintotest3/extra_param_test.go`

The deterministic matrix is:

- merge-planning runtime slots: `4`
- write-and-ingest runtime slots: `1`
- overlap threshold at merge planning: `1000`
- overlap threshold at write-and-ingest planning: `250`
- data group overlap: `500`
- index group 1 overlap: `100`
- index group 2 overlap: `1500`

Under this matrix, the data group skips merge at 4 slots but requires merge at 1 slot. `index 1` skips at both slot counts, and `index 2` merges at both slot counts.

The following commands reproduce the bug on current master with that probe:

```bash
go test ./pkg/dxf/importinto -run '^TestAINativeMergeDecisionDriftMustNotDropKVGroup$' -count=1
go test --tags=intest ./pkg/dxf/importinto -run '^TestSchedulerExtGlobalSort$' -count=1
go test --tags=intest ./tests/realtikvtest/importintotest3 \
-run '^TestImportInto/TestAINativeRuntimeDownscaleMustNotDropDataKVGroup$' \
-count=1 -with-real-tikv -timeout=10m
```

### 2. What did you expect to see? (Required)

Once merge planning finishes, write-and-ingest should preserve every encoded data/index KV group that belongs to that completed plan, even if runtime slots later change.

If a later checksum detects a problem, the import must still fail closed without leaving durable ghost unique-index keys behind.

### 3. What did you see instead? (Required)

The same root reproduces at three levels:

1. Component RED:

```text
expected groups: data, 1, 2
actual groups: 1, 2
```

2. Scheduler RED with one real merge subtask:

```text
merge planning skipped data and index 1 at 4 slots
merge planning emitted one real merge subtask for index 2
write-and-ingest reloaded MaxRuntimeSlots=1
actual groups: 1, 2
missing group: data
```

3. real-TiKV RED:

```text
[Lightning:Restore:ErrChecksumMismatch]
checksum: 0 vs 691282985088154655
total_kvs: 4 vs 6
total_bytes: 144 vs 224
```

The durable consequence is worse than a status-only checksum error. After the failed import:

```text
SELECT * FROM t: empty
ADMIN CHECK TABLE t: data inconsistency
INSERT INTO t VALUES (10, 101, 999, 'new'): Duplicate entry
```

So the base-row KV group is missing, while the unique-index KVs remain in TiKV and reject a later valid write.

The counterfactual is GREEN when write-and-ingest treats the completed merge outputs as authoritative and copies every encoded KV group absent from that map. Under that single change, component, scheduler, and real-TiKV runs all pass; rows are present, unique indexes match, and `ADMIN CHECK TABLE` passes.

#### Likely root cause

`generateMergeSortSpecs` decides whether each KV group skips merge according to the current runtime slots. Only the groups chosen to merge produce merge outputs.

Later, `getSortedKVMetasForIngest` rebuilds the ingest KV-group set by re-running `skipMergeSort(..., planCtx.ThreadCnt)` under the new runtime slot count instead of consuming the completed merge output map as decision provenance.

After a `4 -> 1` downscale:

- `data` flips from skip to merge,
- `index 1` stays skip,
- `index 2` stays merge.

But there is no merge output for `data`, because the completed merge plan had intentionally skipped it. Since write-and-ingest also does not fall back to the encode metadata for that group, `data` disappears from the ingest set entirely.

This is distinct from `#70372`: that issue depends on conflict cleanup after functional-index schema evolution. This root does not require schema evolution or conflict capture; it is a cross-stage decision-provenance bug between merge planning and ingest planning.

### 4. What is your TiDB version? (Required)

```text
Git Commit Hash: 8b62b76214af40eed6aeac045948e2d53adbb7e3
Store: tikv
```

The persistent consequence was validated with one real nightly TiKV plus fake GCS-backed Global Sort storage. The scheduler/component reproductions hit the same current-master logic without requiring a naturally huge file set.

### Impact

This can be triggered by a supported operational control during a long-running Global Sort import. The import reports checksum mismatch only after unique-index SSTs have already been ingested. The table can be left empty while ghost unique-index keys still reject later valid inserts. Recovery then requires explicit repair or index rebuild; it is not a clean rollback.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.