pingcap / pingcap/tidb

IMPORT INTO nextgen: add Init->Prepared async prepare flow (prepare_mode)

Open
#68,580 0 comments 0 reactions 0 assignees View on GitHub
component/DXF component/import type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

### Problem
`IMPORT INTO` can spend more time in submit-path prepare work before users see an actionable task lifecycle when the dataset is large, and some heavy initialization work may be duplicated between submit and later planning.

Data Size (Source_File_Size) | Prepare Time
-- | --
10.21GiB | 1.176 sec
80.54GiB | 1.172 sec
181.1GiB | 1.312 sec
906.2GiB | 1.681 sec
8.067TiB | 7.087 sec
27.84TiB | 18.227 sec

### Proposal
Introduce a framework-level `Init -> Prepared` flow guarded by `prepare_mode` for nextgen `IMPORT INTO` (global-sort path in v1):

1. Task starts as `pending + init` with `prepare_mode=required`.
2. Framework runs prepare logic asynchronously while the task is still `pending + init`.
3. On prepare success, framework atomically persists:
- `step=prepared` (new marker step)
- updated task `meta`
- updated `concurrency` / `max_node_count`
- updated `extra_params`
4. Scheduler continues normal business-step progression from `prepared` to the first import step.

### User-visible behavior
1. `SHOW IMPORT JOBS` shows phase `preparing` for prepare-enabled jobs.
2. Prepare-heavy work is moved out of the synchronous submit critical path.
3. Resource-related parameters computed during prepare are persisted before first business-step subtask scheduling.
4. Compatibility default remains unchanged (`prepare_mode=disabled` keeps existing flow).

### Scope (v1)
1. Nextgen `IMPORT INTO` global-sort path only.
2. Framework support for `StepPrepared` and pending-step transition API.
3. Import scheduler/planner integration with prepared external metadata consumption.

### Alternatives considered
1. `Init -> Prepared` (chosen): smaller blast radius, explicit durable checkpoint, no phase mutation in `extra_params`.
2. `Init -> Preparing -> Prepared` (deferred): better in-table progress observability, but higher scheduler/storage/test complexity for current one-shot prepare scope.

### Acceptance criteria
1. Prepare-enabled DXF tasks follow `StepInit -> StepPrepared -> first business step`.
2. A dedicated pending-step transition persists prepared data atomically and tolerates benign zero-row CAS races.
3. Import jobs enter `running + preparing` during prepare, then transition to existing phase flow at first business step.
4. Targeted framework/import tests cover HA/race/cancel paths and pass in Ready-profile validation.

### Risks / compatibility notes
1. While prepare is running, task table step remains `init` (in-table observability tradeoff for v1).
2. Some file-discovery-dependent failures become asynchronous failures in `preparing`.
3. External prepared metadata may produce orphan attempt artifacts under retries; cleanup should rely on task-scoped prefix lifecycle handling.

### Reference
Design/implementation plan source:
- `docs/agents/import-into/execplan-nextgen-init-prepared.md`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.