pingcap / pingcap/tidb

gc not working due to "scanRegion from PD failed ... grpc: received message larger than max (5369472 vs. 4194304) "

Open
#43,677 0 comments 0 reactions 0 assignees View on GitHub
type/question
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
0. TiDB cluster with config max-index-length = 12288
1. Use lightning local backend to import single table with 10+ TB of csv, table schema with 13 index, some of them are as large as 10k, e.g. KEY `xx` ((cast(json_extract(`xx`, _utf8mb4'$') as char(10000) array))).

### 2. What did you expect to see? (Required)
1. GC should work correctly with lightning imported data

### 3. What did you see instead (Required)
1. Lightning import is OK, but GC stops working
![image](https://github.com/pingcap/tidb/assets/7403864/057a6ee9-0481-4f34-b744-425a75015692)

``` tidb.log
[2023/05/09 12:38:41.831 +00:00] [WARN] [backoff.go:158] ["pdRPC backoffer.maxSleep 40000ms is exceeded, errors:\nscanRegion from PD failed, startKey:...., limit: '\\u0080', err: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5369472 vs. 4194304) at 2023-05-09T12:38:40.210860485Z\nlongest sleep type: pdRPC, time: 41444ms"]
...
[2023/05/09 12:38:41.833 +00:00] [ERROR] [gc_worker.go:1214] ["[gc worker] resolve locks failed"] [uuid=620002f9dc40010] [safePoint=441354886899826688] [error=]
[2023/05/09 12:38:41.833 +00:00] [ERROR] [gc_worker.go:769] ["[gc worker] resolve locks returns an error"] [uuid=620002f9dc40010] [error=]
[2023/05/09 12:38:41.833 +00:00] [ERROR] [gc_worker.go:234] ["[gc worker] runGCJob"] [error=]
```
### 4. What is your TiDB version? (Required)
Release Version: v7.0.0
Edition: Community
Git Commit Hash: 7376954cd868dbc44fc3015c9ef89c53749339a7
Git Branch: heads/refs/tags/v7.0.0
UTC Build Time: 2023-03-29 13:32:13
GoVersion: go1.20.2
Race Enabled: false
TiKV Min Version: 6.2.0-alpha
Check Table Before Drop: false
Store: unistore

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.