pingcap / pingcap/tidb

global sort: lowering runtime slots can invalidate persisted merge outputs and ingest plans

Open
#70,408 0 comments 0 reactions 0 assignees View on GitHub
affects-9.0 classic component/ddl component/DXF component/global-sort component/import may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/moderate type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

### 1. Minimal reproduce step (Required)

This reproduction only reduces concurrency, and both the old and new values are at most 16.

1. Run a global-sort distributed ADD INDEX on one eligible `16c/32 GiB` node with effective runtime slots set to 16.
2. Let merge-sort planning process a KV group containing 1,000,000 overlapping data files. At concurrency 16, the planner accepts and persists 250 merge groups of 4,000 input files.

#### A. Reduce concurrency while merge sort is pending or running

1. Before all merge-sort subtasks finish, execute `ADMIN ALTER DDL JOBS THREAD = 4`.
2. A pending subtask that starts with concurrency 4 still splits its 4,000 input files into `max(ceil(4000 / 250), 4) = 16` merge shares because one thread may merge at most 250 files.
3. A running subtask was already split into 16 shares. Its resource-modification path tunes the worker-pool size but does not repartition those shares.
4. Therefore, regardless of how many subtasks were pending or running at the time of the reduction, the 250 persisted groups still produce `250 * 16 = 4,000` target files. The adjusted target for concurrency 4 is only 1,000.

Deterministic merge-output-count reproduction

```go
// Run inside pkg/ingestor/globalsort so splitDataFiles is visible.
groups, err := DivideMergeSortDataFiles(make([]string, 1_000_000), 1, 16)
require.NoError(t, err)
require.Len(t, groups, 250)

plannedTargetFiles := 0
reducedRuntimeTargetFiles := 0
for _, group := range groups {
require.Len(t, group, 4_000)
plannedShares := splitDataFiles(group, 16)
reducedRuntimeShares := splitDataFiles(group, 4)
require.Len(t, plannedShares, 16)
require.Len(t, reducedRuntimeShares, 16)
plannedTargetFiles += len(plannedShares)
reducedRuntimeTargetFiles += len(reducedRuntimeShares)
}
require.Equal(t, 4_000, plannedTargetFiles)
require.Equal(t, plannedTargetFiles, reducedRuntimeTargetFiles)

reducedTarget := int(simplesst.GetAdjustedMergeSortOverlapThreshold(4))
require.Equal(t, 1_000, reducedTarget)
require.Greater(t, reducedRuntimeTargetFiles, reducedTarget)
```

The reduction does not produce more files than the original concurrency-16 plan. It makes the unchanged 4,000-file result exceed the new concurrency-4 target by four times.

#### B. Reduce concurrency after merge sort or during ingest

1. Alternatively, let merge sort finish at concurrency 16 and produce the 4,000 target files.
2. Let the write-and-ingest step persist its subtask metadata. With one ingest instance, a subtask covering the KV group can reference those merge outputs together with range-job boundaries calculated for the original resource envelope.
3. Before that ingest subtask runs, or while it is running, execute `ADMIN ALTER DDL JOBS THREAD = 4`.
4. The existing `DataFiles`, `StatFiles`, key range, and range-job boundaries are not regenerated, but the ingest worker concurrency and memory capacity are changed to the 4-slot resource.

Lowering the concurrency cannot reduce the number of files that already exist.

The resource reduction also shrinks the ingest memory envelope. On a `16c/32 GiB` node, changing from 16 slots to 4 reduces the task memory capacity from approximately 32 GiB to 8 GiB. The external engine memory limit therefore falls from approximately 14.8 GiB to 3.7 GiB (`memCapacity / 6.5 * 3`), while the persisted ingest work is unchanged.

This matters most for a hot or skewed key range that intersects many of the merge outputs. `readAllData` can create up to 1,000 concurrent file readers and accumulates the selected range data in memory. More files also increase range-size estimation error and per-file buffer/KV overhead. Work that fit the 16-slot envelope can therefore fail after the reduction with `ErrCannotAcquireMemory` or another reader/resource error. The source already lowers the overlap threshold below concurrency 8 specifically because the smaller load-memory budget can otherwise block on the memory limiter.

IMPORT INTO has the same persisted-plan problem. Its effective runtime slots can be reduced through DXF `MaxRuntimeSlots`; a retried or restarted executor then uses the smaller current resource while the existing merge outputs and write-and-ingest subtask metadata remain unchanged.

NextGen is not affected by this user-triggered scenario today. Its ADD INDEX resource parameters are calculated automatically, and NextGen does not support users changing the concurrency parameter at runtime, so users cannot trigger the `16 -> 4` reduction described above. The current behavior is therefore acceptable for NextGen. The same compatibility invariant would be needed if user-controlled runtime concurrency changes are supported there in the future.

### 2. What did you expect to see? (Required)

Every accepted runtime-slot reduction should preserve this invariant:

> Existing intermediate files and every persisted or running subtask remain executable within the new resource envelope.

The implementation may reject or defer an unsafe reduction, retain a minimum resource floor for artifacts that already exist, regenerate/repartition pending work (including re-merging files when necessary), or make ingest adaptively process the existing files with bounded readers and memory. The issue does not require one specific mechanism.

Persisting or pinning only the planning concurrency or predicted target-file count is not a complete solution. It can keep the merge output consistent with the original plan or detect a mismatch, but it still leaves up to 4,000 files and ingest subtasks sized for 16 slots for a 4-slot ingest executor to process.

### 3. What did you see instead (Required)

The current implementation applies a live resource reduction to the executor without validating that persisted artifacts and subtasks are compatible with the smaller resource:

- If concurrency is reduced during merge sort, every persisted 4,000-file group still has 16 shares, so the step produces 4,000 target files even though the adjusted concurrency-4 target is 1,000.
- If concurrency is reduced after merge sort, the 4,000 existing target files remain while the adjusted concurrency-4 target is 1,000.
- The persisted ingest file lists and range boundaries are not regenerated.
- The ingest executor receives the smaller worker concurrency and memory capacity, so a previously valid range batch can exceed its new reader or memory budget.
- More target files increase estimation deviation and the possible imbalance in files and data assigned to ingest ranges, increasing the chance that an individual range no longer fits.

As a result, a valid ADD INDEX or IMPORT INTO operation can fail after a supported runtime concurrency reduction.

### 4. What is your TiDB version? (Required)

Current `master`:

```text
Git Commit Hash: 94abd9fa63fdd1cc541eb7d32ef11b201ded22d6
Git Branch: master
Nearest tag: v9.0.0-beta.2.pre
```

Contributor guide

Open the contributing guide

Research direction

Start in pkg/ingestor/globalsort with DivideMergeSortDataFiles, splitDataFiles, and simplesst.GetAdjustedMergeSortOverlapThreshold, then trace the runtime-slot reduction path for ADD INDEX and IMPORT INTO. Compare persisted merge outputs and ingest subtask metadata with the reduced worker and memory capacity. Done means an accepted reduction either remains executable for existing work or is safely rejected, deferred, or adapted without invalidating persisted artifacts.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.