importinto: support batch import multiple tables from files named in `dumpling` style
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Feature Request
**Is your feature request related to a problem? Please describe:**
**Describe the feature you'd like:**
With [import into](https://github.com/pingcap/tidb/issues/42930), we can now import into a single table using physical mode, but in many cases user will dump data using `dumpling`, then import, and there're many tables. Those files are named in [this format](https://docs.pingcap.com/tidb/stable/dumpling-overview#format-of-exported-files). It would helpful if we import them together using import into statement too.
the syntax of importing multiple tables from files named in [`dumpling` style](https://docs.pingcap.com/tidb/stable/dumpling-overview#format-of-exported-files) is very similar with import into a single table:
```
IMPORT INTO * FROM WITH table_filter = 'test.A*,foo.X', thread = 8 ...
```
The `*` in the statement is just a indicator to mean we are importing multiple tables.
This statement works as a syntax sugar for normal `import into` that only imports into single table, it works like this:
- walk through `` to get all matched files, and from file name we can get which table they are targeting to
- filter tables to get those we should import
- create tables if not exists.
- create a separate `import into` job for each target table with a thread <= `thread` param specified in the statement
- all those jobs are run in `detached` mode
- after all those jobs created, we return a summary and a batch id(all those jobs have the same batch) to query progress of this multiple-table-import. the summary(might be changed) is something like:
```
Total data file size: 1.2 TiB, 100 jobs created. Batch ID: xxx.
```
With the batch id, we can get the progress of this import job batch like:
```mysql
mysql> SHOW IMPORT JOB BATCH 'xxx'
Progress: 10/100 jobs finished, 16 jobs running, 3 jobs failed, 1 jobs cancelled, 123G/1.2T data imported.
```
you can also query jobs contained in this batch by
```
SHOW IMPORT JOBS where batch = 'xxx'
```
to cancel the batch, use
```
CANCEL IMPORT JOB BATCH xxx
```
it will cancel all jobs that hasn't done yet, i.e. pending or running.
we only support S3/GCS as `` temporarily, and [tidb_enable_dist_task](https://docs.pingcap.com/tidb/stable/system-variables#tidb_enable_dist_task-new-in-v710) must be enabled to run this sql.
user cannot specified `FORMAT xx` clause explicitly, we will walk all files in `` and determine it's format from the suffix, and only tables matches with `filter` is imported.
The `thread` param only indicate the max cpu usage when importing each table, and as all those jobs runs on [TiDB Distributed eXecution Framework (DXF)](https://docs.pingcap.com/tidb/stable/tidb-distributed-execution-framework), the actual cpu usage depends how many resource managed by DXF, how many tables to import and the rules we used to assign thread to jobs.
## Tasks
- [ ] support where clause in `show import jobs`
- [ ] show more job detail in `show import job(s)`, such as all options, thread, whether it's distributed, global sort or not, etc.
- [ ] lightning code refactor to reuse in batch import
- [x] https://github.com/pingcap/tidb/pull/52147
- [x] https://github.com/pingcap/tidb/pull/52334
- [ ] https://github.com/pingcap/tidb/pull/52469
- [ ] batch import into planner & executer part
- [ ] permission check, user might have `CREATE` permission, but might not have WRITE permission to the target table.
- [ ] show batch import into planner & executer part
- [ ] cancel batch import into planner & executer part
**Describe alternatives you've considered:**
lightning already support this, but import into works as a SQL, more user friendly, and integrated with global sort, so we would to have to feature in import into too.
**Teachability, Documentation, Adoption, Migration Strategy:**
Contributor guide
Assessment
This issue has not been assessed yet.