gc not working due to "scanRegion from PD failed ... grpc: received message larger than max (5369472 vs. 4194304) "
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
0. TiDB cluster with config max-index-length = 12288
1. Use lightning local backend to import single table with 10+ TB of csv, table schema with 13 index, some of them are as large as 10k, e.g. KEY `xx` ((cast(json_extract(`xx`, _utf8mb4'$') as char(10000) array))).
### 2. What did you expect to see? (Required)
1. GC should work correctly with lightning imported data
### 3. What did you see instead (Required)
1. Lightning import is OK, but GC stops working

``` tidb.log
[2023/05/09 12:38:41.831 +00:00] [WARN] [backoff.go:158] ["pdRPC backoffer.maxSleep 40000ms is exceeded, errors:\nscanRegion from PD failed, startKey:...., limit: '\\u0080', err: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5369472 vs. 4194304) at 2023-05-09T12:38:40.210860485Z\nlongest sleep type: pdRPC, time: 41444ms"]
...
[2023/05/09 12:38:41.833 +00:00] [ERROR] [gc_worker.go:1214] ["[gc worker] resolve locks failed"] [uuid=620002f9dc40010] [safePoint=441354886899826688] [error=]
[2023/05/09 12:38:41.833 +00:00] [ERROR] [gc_worker.go:769] ["[gc worker] resolve locks returns an error"] [uuid=620002f9dc40010] [error=]
[2023/05/09 12:38:41.833 +00:00] [ERROR] [gc_worker.go:234] ["[gc worker] runGCJob"] [error=]
```
### 4. What is your TiDB version? (Required)
Release Version: v7.0.0
Edition: Community
Git Commit Hash: 7376954cd868dbc44fc3015c9ef89c53749339a7
Git Branch: heads/refs/tags/v7.0.0
UTC Build Time: 2023-03-29 13:32:13
GoVersion: go1.20.2
Race Enabled: false
TiKV Min Version: 6.2.0-alpha
Check Table Before Drop: false
Store: unistore
Contributor guide
Assessment
This issue has not been assessed yet.