pingcap / pingcap/tidb

resource control inside dist framework

Open
#49,008 0 comments 0 reactions 0 assignees View on GitHub
type/feature-request
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Feature Request

**Is your feature request related to a problem? Please describe:**

- Currently, the dispatcher manager supports up to 4 tasks, and the Scheduler manager can process up to 4 tasks.
- The dispatcher/Scheduler manager does not consider the concurrent configuration of tasks when processing tasks, and resources may be overused, resulting in OOM.
- The above manager does not consider the order of tasks. After task A is processed halfway, it may start processing other tasks due to failover and other reasons, causing task A to keep waiting.

**Describe the feature you'd like:**

This feature will add control over cpu/mem/disk within the nodes managed by framework(see `tidb_service_scope`), improves resource usage, avoids overall resource usage exceeding limits, and ensures task execution order. When executing multiple tasks in parallel, ensure that each task can run normally without affecting each other.
in first version, we consider dist framework can use all resources that it runs on.

### resource control
- [x] #49019
- [x] #49100
- [x] #49135
- [x] https://github.com/pingcap/tidb/pull/49403
- [x] https://github.com/pingcap/tidb/pull/49623
- [x] https://github.com/pingcap/tidb/pull/49758
- [x] https://github.com/pingcap/tidb/pull/49713
- [x] https://github.com/pingcap/tidb/pull/49875
- [x] https://github.com/pingcap/tidb/pull/50091
- [x] https://github.com/pingcap/tidb/pull/50266
- [x] https://github.com/pingcap/tidb/pull/50475
- [ ] add-index restrict resource usage
- [x] import-into restrict resource usage and allow multiple jobs running concurrently
- [x] https://github.com/pingcap/tidb/pull/51335
- [x] https://github.com/pingcap/tidb/pull/51486
- [x] https://github.com/pingcap/tidb/pull/51575
- [x] https://github.com/pingcap/tidb/pull/51776
- [x] https://github.com/pingcap/tidb/pull/50650
- [x] https://github.com/pingcap/tidb/pull/50540
- [x] https://github.com/pingcap/tidb/pull/50724
- [x] https://github.com/pingcap/tidb/pull/50801
- [x] https://github.com/pingcap/tidb/pull/50962
- [x] https://github.com/pingcap/tidb/pull/51136
- [x] https://github.com/pingcap/tidb/pull/51182
- [x] https://github.com/pingcap/tidb/pull/51231
- [x] https://github.com/pingcap/tidb/pull/51022
- [ ] https://github.com/pingcap/tidb/pull/51003
- [x] https://github.com/pingcap/tidb/pull/50987
- [x] https://github.com/pingcap/tidb/pull/51241
- [x] https://github.com/pingcap/tidb/pull/51426
- [x] https://github.com/pingcap/tidb/pull/51662

### others
- [x] https://github.com/pingcap/tidb/pull/49025
- [x] #49125
- [x] https://github.com/pingcap/tidb/issues/49617
- [x] https://github.com/pingcap/tidb/pull/49805
- [x] https://github.com/pingcap/tidb/pull/50224
- [x] https://github.com/pingcap/tidb/pull/50550
- [x] https://github.com/pingcap/tidb/pull/50565
- [x] https://github.com/pingcap/tidb/pull/50664
- [x] https://github.com/pingcap/tidb/pull/50868
- [x] https://github.com/pingcap/tidb/pull/50912
- [x] https://github.com/pingcap/tidb/pull/51279
- [x] https://github.com/pingcap/tidb/pull/51071

**Describe alternatives you've considered:**

**Teachability, Documentation, Adoption, Migration Strategy:**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.