[import] Global Sort IMPORT can omit a KV group after runtime downscale and leave ghost unique indexes
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
# [import] Global Sort IMPORT can omit a KV group after runtime downscale and leave ghost unique indexes
## Bug Report
### 1. Minimal reproduce step (Required)
This issue needs three ordinary ingredients:
1. Global Sort `IMPORT INTO` without forced merge.
2. A completed merge plan where one KV group skips merge at 4 runtime slots, but would require merge at 1 slot.
3. A supported `max_runtime_slots` downscale during a non-trivial merge, followed by a TiDB restart or task retry before write-and-ingest planning reloads task metadata.
The production-shaped sequence is:
1. Start a large Global Sort import.
2. Let merge planning run while `GetRuntimeSlots()` is 4, so one KV group is intentionally left in the encode output and another KV group still goes through a real merge subtask.
3. During that merge stage, use the documented throttle:
```text
POST /dxf/task/{taskID}/max_runtime_slots?value=1
```
4. Before write-and-ingest planning, let TiDB restart or let the task retry so the next planning context reloads `MaxRuntimeSlots=1`.
After the reload, write-and-ingest re-evaluates the old skip/merge predicate under the new slot count instead of consuming the completed merge output map.
For maintainers, the current-master behavior was made deterministic with regression probes around:
- `pkg/dxf/importinto/planner.go`
- `pkg/dxf/importinto/scheduler.go`
- `pkg/dxf/importinto/planner_test.go`
- `pkg/dxf/importinto/scheduler_testkit_test.go`
- `tests/realtikvtest/importintotest3/extra_param_test.go`
The deterministic matrix is:
- merge-planning runtime slots: `4`
- write-and-ingest runtime slots: `1`
- overlap threshold at merge planning: `1000`
- overlap threshold at write-and-ingest planning: `250`
- data group overlap: `500`
- index group 1 overlap: `100`
- index group 2 overlap: `1500`
Under this matrix, the data group skips merge at 4 slots but requires merge at 1 slot. `index 1` skips at both slot counts, and `index 2` merges at both slot counts.
The following commands reproduce the bug on current master with that probe:
```bash
go test ./pkg/dxf/importinto -run '^TestAINativeMergeDecisionDriftMustNotDropKVGroup$' -count=1
go test --tags=intest ./pkg/dxf/importinto -run '^TestSchedulerExtGlobalSort$' -count=1
go test --tags=intest ./tests/realtikvtest/importintotest3 \
-run '^TestImportInto/TestAINativeRuntimeDownscaleMustNotDropDataKVGroup$' \
-count=1 -with-real-tikv -timeout=10m
```
### 2. What did you expect to see? (Required)
Once merge planning finishes, write-and-ingest should preserve every encoded data/index KV group that belongs to that completed plan, even if runtime slots later change.
If a later checksum detects a problem, the import must still fail closed without leaving durable ghost unique-index keys behind.
### 3. What did you see instead? (Required)
The same root reproduces at three levels:
1. Component RED:
```text
expected groups: data, 1, 2
actual groups: 1, 2
```
2. Scheduler RED with one real merge subtask:
```text
merge planning skipped data and index 1 at 4 slots
merge planning emitted one real merge subtask for index 2
write-and-ingest reloaded MaxRuntimeSlots=1
actual groups: 1, 2
missing group: data
```
3. real-TiKV RED:
```text
[Lightning:Restore:ErrChecksumMismatch]
checksum: 0 vs 691282985088154655
total_kvs: 4 vs 6
total_bytes: 144 vs 224
```
The durable consequence is worse than a status-only checksum error. After the failed import:
```text
SELECT * FROM t: empty
ADMIN CHECK TABLE t: data inconsistency
INSERT INTO t VALUES (10, 101, 999, 'new'): Duplicate entry
```
So the base-row KV group is missing, while the unique-index KVs remain in TiKV and reject a later valid write.
The counterfactual is GREEN when write-and-ingest treats the completed merge outputs as authoritative and copies every encoded KV group absent from that map. Under that single change, component, scheduler, and real-TiKV runs all pass; rows are present, unique indexes match, and `ADMIN CHECK TABLE` passes.
#### Likely root cause
`generateMergeSortSpecs` decides whether each KV group skips merge according to the current runtime slots. Only the groups chosen to merge produce merge outputs.
Later, `getSortedKVMetasForIngest` rebuilds the ingest KV-group set by re-running `skipMergeSort(..., planCtx.ThreadCnt)` under the new runtime slot count instead of consuming the completed merge output map as decision provenance.
After a `4 -> 1` downscale:
- `data` flips from skip to merge,
- `index 1` stays skip,
- `index 2` stays merge.
But there is no merge output for `data`, because the completed merge plan had intentionally skipped it. Since write-and-ingest also does not fall back to the encode metadata for that group, `data` disappears from the ingest set entirely.
This is distinct from `#70372`: that issue depends on conflict cleanup after functional-index schema evolution. This root does not require schema evolution or conflict capture; it is a cross-stage decision-provenance bug between merge planning and ingest planning.
### 4. What is your TiDB version? (Required)
```text
Git Commit Hash: 8b62b76214af40eed6aeac045948e2d53adbb7e3
Store: tikv
```
The persistent consequence was validated with one real nightly TiKV plus fake GCS-backed Global Sort storage. The scheduler/component reproductions hit the same current-master logic without requiring a naturally huge file set.
### Impact
This can be triggered by a supported operational control during a long-running Global Sort import. The import reports checksum mismatch only after unique-index SSTs have already been ingested. The table can be left empty while ghost unique-index keys still reject later valid inserts. Recovery then requires explicit repair or index rebuild; it is not a clean rollback.
Contributor guide
Assessment
This issue has not been assessed yet.