Restore: new pipeline to improve performance
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
Currently we have a fixed goroutine pool to download and ingest SST files. and we need download 3 replicas but ingest only 1 replica. due to ingest happened after download. we have a fluctuation during restoration.

This is because the **unbalanced restore requests** and **unfair restore goroutine scheduler**.

To Solve **unbalanced restore requests**. we can introduce the token bucket for rate limiting, to ensure br send requests evenly on each node.
The code prototype looks like:
```
type tokenForEachNode chan struct{}
func (token tokenForEachNode) getToken() {
_ = <-token
}
func (token tokenForEachNode) backToken() {
token <- struct{}{}
}
type STORE_ID_T int64
var tokensMap map[STORE_ID_T]tokenForEachNode
// If we specify the size of the channel,
// it is equivalent to specify the concurrency to request TiKV.
for _, storeID := range storeIDs {
tokenCh := make(tokenForEachNode, concurrency)
for i := 0; i < concurrency; i += 1 {
tokenCh.backToken()
}
}
```
To Solve **unfair restore goroutine scheduler**. we need introduce a new model to pipeline downloads/ingests. Make the interior more orderly.
Use this issue to trace the discussion.
Contributor guide
Assessment
This issue has not been assessed yet.