pingcap / pingcap/tidb

add index failed with error “Error 1105 (HY000): etcdserver: request timed out, possibly due to connection lost” when inject network partition between pd leader and pd follower

Open
#48,862 1 comment 0 reactions 1 assignee Claimed by @ywqzzy View on GitHub
affects-7.6 affects-8.1 component/ddl may-affects-5.3 may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 severity/moderate type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
tidb_enable_dist_task='on'
1、run workload
2、add index for one table
3、inject network partition between pd leader and pd follower(pd can elect new leader quickly)

case config:
tag: "add-index-dist-local-sort"
workloads:
- type: sysbench
storage: s3://benchmark/sysbench_64_7000w
db: sysbench_64_7000w
tableNum: 64
tableSize: 70000000
threads: 64
subType: oltp_read_write
ignoreErrors: "all"
cases:
- name: "ha_pdleader_to_pdfollower_all_network_partition"
warmUpTime: 1m
faultType: network_partition
selector: "pdleader_to_pdfollower(all)"
period: "@every 5m"
faultDuration: 2m
faultTotalRunTime: 30m
ddl:
addIndex:
- db: "sysbench_64_7000w"
table: "sbtest1"
timeout: 60m
times: 5
index_col_name: "c"
job_type: "ingest"
admin_check: true
count_check: false

### 2. What did you expect to see? (Required)
add index can success

### 3. What did you see instead (Required)
add index failed with error “Error 1105 (HY000): etcdserver: request timed out, possibly due to connection lost” when inject network partition between pd leader and pd follower

add index failed at 2023-11-23 15:02:29 (Error 1105 (HY000): etcdserver: request timed out, possibly due to connection lost)
operatorLogs:
[2023-11-23 14:59:18] ###### start adding index
alter table sbtest1 add index index_test_1700722758708 (c)
[2023-11-23 14:59:18] ###### wait for ddl job finish

### 4. What is your TiDB version? (Required)
git hash:2b88d0420608654a20bd16faaf46600e8b7cc837

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.