pingcap / pingcap/tidb

importinto: add SHOW RAW IMPORT JOB(S) for machine-friendly job stats

Open
#66,461 0 comments 0 reactions 0 assignees View on GitHub
component/import component/lightning type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

### Motivation

`SHOW IMPORT JOB(S)` is primarily human-oriented (formatted strings, evolving column set). Import SDK / Lightning's import-into backend needs a stable, machine-friendly contract to poll job status/progress and compute ETA without parsing logs or brittle table output.

Related: #61088 improves the human-readable progress of `SHOW IMPORT JOBS`; this issue focuses on a stable API surface for programmatic consumers.

### Proposal

Add new statements:

- `SHOW RAW IMPORT JOB `
- `SHOW RAW IMPORT JOBS [WHERE group_key = '']`

Return a small, stable schema:

| Job_ID | Group_Key | Raw_Stats |

`Raw_Stats` is a JSON contract (keep it stable; evolve additively). Suggested fields (example):

```json
{
"job_id": 123,
"group_key": "g",
"data_source": "s3://bucket/path/*.csv?...",
"target_table": "`db`.`tbl`",
"table_id": 112,
"phase": "importing",
"status": "running",
"source_file_size_bytes": 123456,
"imported_rows": 1000,
"current_step": {
"name": "ingest",
"processed_bytes": 123,
"total_bytes": 456,
"speed_bytes_per_sec": 789,
"remaining_seconds": 10
},
"create_time_unix": 1700000000,
"start_time_unix": 1700000001,
"update_time_unix": 1700000002,
"end_time_unix": 0,
"error_message": ""
}
```

### Semantics / compatibility

- Visibility should match `SHOW IMPORT JOB(S)`: SUPER can see all jobs; otherwise only jobs created by the current user.
- `WHERE group_key = ...` is the intended polling pattern for import-into job groups.

### Implementation (WIP)

WIP branch: `rawImportInto` (parser/planner/executor + importsdk/lightning consumption + tests). I can follow up with a PR once the syntax/contract is agreed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.