pingcap / pingcap/tidb

Restore: new pipeline to improve performance

Open
#49,886 0 comments 0 reactions 0 assignees View on GitHub
component/br type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

Currently we have a fixed goroutine pool to download and ingest SST files. and we need download 3 replicas but ingest only 1 replica. due to ingest happened after download. we have a fluctuation during restoration.

![image](https://github.com/pingcap/tidb/assets/5906259/fbb99e5c-b6c4-4d95-a525-8bbcdc4b69cc)

This is because the **unbalanced restore requests** and **unfair restore goroutine scheduler**.
![image](https://github.com/pingcap/tidb/assets/5906259/29492fda-b20e-44ef-83ac-d23a1ba56f97)

To Solve **unbalanced restore requests**. we can introduce the token bucket for rate limiting, to ensure br send requests evenly on each node.

The code prototype looks like:
```
type tokenForEachNode chan struct{}

func (token tokenForEachNode) getToken() {
_ = <-token
}

func (token tokenForEachNode) backToken() {
token <- struct{}{}
}

type STORE_ID_T int64
var tokensMap map[STORE_ID_T]tokenForEachNode

// If we specify the size of the channel,
// it is equivalent to specify the concurrency to request TiKV.
for _, storeID := range storeIDs {
tokenCh := make(tokenForEachNode, concurrency)
for i := 0; i < concurrency; i += 1 {
tokenCh.backToken()
}
}
```

To Solve **unfair restore goroutine scheduler**. we need introduce a new model to pipeline downloads/ingests. Make the interior more orderly.

Use this issue to trace the discussion.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.