resource control inside dist framework
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Feature Request
**Is your feature request related to a problem? Please describe:**
- Currently, the dispatcher manager supports up to 4 tasks, and the Scheduler manager can process up to 4 tasks.
- The dispatcher/Scheduler manager does not consider the concurrent configuration of tasks when processing tasks, and resources may be overused, resulting in OOM.
- The above manager does not consider the order of tasks. After task A is processed halfway, it may start processing other tasks due to failover and other reasons, causing task A to keep waiting.
**Describe the feature you'd like:**
This feature will add control over cpu/mem/disk within the nodes managed by framework(see `tidb_service_scope`), improves resource usage, avoids overall resource usage exceeding limits, and ensures task execution order. When executing multiple tasks in parallel, ensure that each task can run normally without affecting each other.
in first version, we consider dist framework can use all resources that it runs on.
### resource control
- [x] #49019
- [x] #49100
- [x] #49135
- [x] https://github.com/pingcap/tidb/pull/49403
- [x] https://github.com/pingcap/tidb/pull/49623
- [x] https://github.com/pingcap/tidb/pull/49758
- [x] https://github.com/pingcap/tidb/pull/49713
- [x] https://github.com/pingcap/tidb/pull/49875
- [x] https://github.com/pingcap/tidb/pull/50091
- [x] https://github.com/pingcap/tidb/pull/50266
- [x] https://github.com/pingcap/tidb/pull/50475
- [ ] add-index restrict resource usage
- [x] import-into restrict resource usage and allow multiple jobs running concurrently
- [x] https://github.com/pingcap/tidb/pull/51335
- [x] https://github.com/pingcap/tidb/pull/51486
- [x] https://github.com/pingcap/tidb/pull/51575
- [x] https://github.com/pingcap/tidb/pull/51776
- [x] https://github.com/pingcap/tidb/pull/50650
- [x] https://github.com/pingcap/tidb/pull/50540
- [x] https://github.com/pingcap/tidb/pull/50724
- [x] https://github.com/pingcap/tidb/pull/50801
- [x] https://github.com/pingcap/tidb/pull/50962
- [x] https://github.com/pingcap/tidb/pull/51136
- [x] https://github.com/pingcap/tidb/pull/51182
- [x] https://github.com/pingcap/tidb/pull/51231
- [x] https://github.com/pingcap/tidb/pull/51022
- [ ] https://github.com/pingcap/tidb/pull/51003
- [x] https://github.com/pingcap/tidb/pull/50987
- [x] https://github.com/pingcap/tidb/pull/51241
- [x] https://github.com/pingcap/tidb/pull/51426
- [x] https://github.com/pingcap/tidb/pull/51662
### others
- [x] https://github.com/pingcap/tidb/pull/49025
- [x] #49125
- [x] https://github.com/pingcap/tidb/issues/49617
- [x] https://github.com/pingcap/tidb/pull/49805
- [x] https://github.com/pingcap/tidb/pull/50224
- [x] https://github.com/pingcap/tidb/pull/50550
- [x] https://github.com/pingcap/tidb/pull/50565
- [x] https://github.com/pingcap/tidb/pull/50664
- [x] https://github.com/pingcap/tidb/pull/50868
- [x] https://github.com/pingcap/tidb/pull/50912
- [x] https://github.com/pingcap/tidb/pull/51279
- [x] https://github.com/pingcap/tidb/pull/51071
**Describe alternatives you've considered:**
**Teachability, Documentation, Adoption, Migration Strategy:**
Contributor guide
Assessment
This issue has not been assessed yet.