pingcap / pingcap/tidb

DXF: support modify task concurrency and business params at runtime

Open
#57,497 0 comments 0 reactions 0 assignees View on GitHub
type/feature-request
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Feature Request

**Is your feature request related to a problem? Please describe:**

Currently, after a task is submitted to DXF, we cannot change the max resource it can be used at runtime. But we need this feature in some cases, such as when submitting alter table xxx add index, current value of [tidb_ddl_reorg_worker_cnt](https://docs.pingcap.com/tidb/stable/system-variables#tidb_ddl_reorg_worker_cnt) of which default is 4, is used as the task concurrency, later users might want to accelerate it by increasing the concurrency, which is a supported feature when adding index in standalone mode. And import-into might need it too.

Also we need to modify some business specific param at runtime, such as `batch-size` and `max-write-speed` for add-index.

**Describe the feature you'd like:**

support modify task concurrency at runtime, we want to make it easier to extend as we might need to modify other task config at runtime later. and support modify business specific param

**Describe alternatives you've considered:**

this is new feature
**Teachability, Documentation, Adoption, Migration Strategy:**

DXF is the underlying framework behind upper business, including add-index and import-into, so to support change concurrency or business specific param for upper business, DXF need to support it first, then upper layer can integrate the ability and expose some API to user

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.