Lightning: Let user control when to post-process tables in parallel-import
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Feature Request
**Is your feature request related to a problem? Please describe:**
Lightning since v5.3 supported parallel import of single table distributed to multiple instances. The post-process step — "checksum" and "analyze" — however can only be performed once the entire table is completed. Lightning currently uses an automatic barrier, ensuring the last instance touching the table is the one that performs post-processing.
```
+-----------+
| A |
+-----------+
+------------+---+
| B | * | (* = post-process)
+------------+---+
+-----+
| C |
+-----+
-------------------------> time
```
However, for this to work, the duration which the Lightning instances processing the table must overlap to a continuous range. If some gap happened, post-process will run multiple times:
```
+-----------+
| A |
+-----------+
+------------+---+
| B | * |
+------------+---+
+-----+---+
| C | * |
+-----+---+
------------------------------------------> time
```
In fact, this problem is [spelled out in the documentation](https://docs.pingcap.com/tidb/stable/tidb-lightning-distributed-import#during-an-import-an-error-target-table-is-calculating-checksum-please-wait-until-the-checksum-is-finished-and-try-again-is-reported).
**Describe the feature you'd like:**
Introduce a pair of commands to manually bracket the duration which a table is considered "importing" in parallel-import, so post-process happened only once at a user-desired moment.
```
+=========================================+---+
| | * |
+=========================================+---+
+-----------+
| A |
+-----------+
+------------+
| B |
+------------+
+-----+
| C |
+-----+
---------------------------------------------------> time
```
Perhaps something like
```sh
./tidb-lightning-ctl -c config.toml -d /data/schemas/ -start-parallel-import=all
./tidb-lightning -c config.toml -d /data/part_01/ &
./tidb-lightning -c config.toml -d /data/part_02/ &
# ...
./tidb-lightning-ctl -c config.toml -d /data/schemas/ -finish-parallel-import='`db`.`tbl`' # do only 1 table
./tidb-lightning-ctl -c config.toml -d /data/schemas/ -finish-parallel-import=all # post-process every other table
```
This also gives a new feature in Lightning, that the user can *sequentially* import parts of a table through multiple invocations, essentially acting as controlled checkpoints.
```
+=========================================+---+
| | * |
+=========================================+---+
+-----+ +-----+ +-------+ +---+
| A | | B | | C | | D |
+-----+ +-----+ +-------+ +---+
---------------------------------------------------> time
```
**Describe alternatives you've considered:**
This is in fact already possible by directly manipulating the ``` `lightning_metadata`.`table_meta` ``` table to deceive other Lightning instances that there is a long-running task, to prevent any of them declaring the table is done before I said so.
The `-start-parallel-import` command is equivalent to:
1. Import the table schemas *only*, do not import any data
2. Obtain the table IDs of every table.
3. `CREATE SCHEMA lightning_metadata; CREATE TABLE task_meta (...); CREATE TABLE table_meta (...);`
4. `INSERT INTO task_meta (task_id, status) VALUES (0, '');`
5. `INSERT TABLE table_meta (task_id, table_id, table_name, status) VALUES (0, 1234, '`db`.`tbl`', '');` (replace 1234 with the table IDs from step 2)
The `-finish-parallel-import` command is equivalent to:
1. `DELETE FROM task_meta WHERE task_id = 0; DELETE FROM table_meta WHERE task_id = 0;`
2. Run `tidb-lightning` proper with a constructed data source that contains a single empty `db.tbl.csv` file for each involved table, to trigger the data import flow, and make itself the barrier breaker, which will then perform the post-process step.
These steps are however pretty error-prone, it's better to encapsulate these implementation details.
**Teachability, Documentation, Adoption, Migration Strategy:**
--
Contributor guide
Assessment
This issue has not been assessed yet.