pingcap / pingcap/tidb

Lightning: Let user control when to post-process tables in parallel-import

Open
#50,940 19 comments 0 reactions 0 assignees View on GitHub
component/lightning type/feature-request
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Feature Request

**Is your feature request related to a problem? Please describe:**

Lightning since v5.3 supported parallel import of single table distributed to multiple instances. The post-process step — "checksum" and "analyze" — however can only be performed once the entire table is completed. Lightning currently uses an automatic barrier, ensuring the last instance touching the table is the one that performs post-processing.

```
+-----------+
| A |
+-----------+
+------------+---+
| B | * | (* = post-process)
+------------+---+
+-----+
| C |
+-----+
-------------------------> time
```

However, for this to work, the duration which the Lightning instances processing the table must overlap to a continuous range. If some gap happened, post-process will run multiple times:

```
+-----------+
| A |
+-----------+
+------------+---+
| B | * |
+------------+---+
+-----+---+
| C | * |
+-----+---+
------------------------------------------> time
```

In fact, this problem is [spelled out in the documentation](https://docs.pingcap.com/tidb/stable/tidb-lightning-distributed-import#during-an-import-an-error-target-table-is-calculating-checksum-please-wait-until-the-checksum-is-finished-and-try-again-is-reported).

**Describe the feature you'd like:**

Introduce a pair of commands to manually bracket the duration which a table is considered "importing" in parallel-import, so post-process happened only once at a user-desired moment.

```
+=========================================+---+
| | * |
+=========================================+---+
+-----------+
| A |
+-----------+
+------------+
| B |
+------------+
+-----+
| C |
+-----+
---------------------------------------------------> time
```

Perhaps something like

```sh
./tidb-lightning-ctl -c config.toml -d /data/schemas/ -start-parallel-import=all
./tidb-lightning -c config.toml -d /data/part_01/ &
./tidb-lightning -c config.toml -d /data/part_02/ &
# ...
./tidb-lightning-ctl -c config.toml -d /data/schemas/ -finish-parallel-import='`db`.`tbl`' # do only 1 table
./tidb-lightning-ctl -c config.toml -d /data/schemas/ -finish-parallel-import=all # post-process every other table
```

This also gives a new feature in Lightning, that the user can *sequentially* import parts of a table through multiple invocations, essentially acting as controlled checkpoints.

```
+=========================================+---+
| | * |
+=========================================+---+
+-----+ +-----+ +-------+ +---+
| A | | B | | C | | D |
+-----+ +-----+ +-------+ +---+
---------------------------------------------------> time
```

**Describe alternatives you've considered:**

This is in fact already possible by directly manipulating the ``` `lightning_metadata`.`table_meta` ``` table to deceive other Lightning instances that there is a long-running task, to prevent any of them declaring the table is done before I said so.

The `-start-parallel-import` command is equivalent to:

1. Import the table schemas *only*, do not import any data
2. Obtain the table IDs of every table.
3. `CREATE SCHEMA lightning_metadata; CREATE TABLE task_meta (...); CREATE TABLE table_meta (...);`
4. `INSERT INTO task_meta (task_id, status) VALUES (0, '');`
5. `INSERT TABLE table_meta (task_id, table_id, table_name, status) VALUES (0, 1234, '`db`.`tbl`', '');` (replace 1234 with the table IDs from step 2)

The `-finish-parallel-import` command is equivalent to:

1. `DELETE FROM task_meta WHERE task_id = 0; DELETE FROM table_meta WHERE task_id = 0;`
2. Run `tidb-lightning` proper with a constructed data source that contains a single empty `db.tbl.csv` file for each involved table, to trigger the data import flow, and make itself the barrier breaker, which will then perform the post-process step.

These steps are however pretty error-prone, it's better to encapsulate these implementation details.

**Teachability, Documentation, Adoption, Migration Strategy:**

--

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.