pingcap / pingcap/tidb

add index failed with error “Error 1105 (HY000): mutex: session is expired” when inject network partition between pdleader and followers

Open
#48,546 2 comments 0 reactions 0 assignees View on GitHub
affects-7.5 affects-8.1 affects-8.5 component/ddl severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
tidb_enable_dist_task='on'
1、run workload
2、add index for one table
3、inject network partition between pdleader and followers or pdleader io delay 1s

tag: "ha-test-add-index"
workloads:
- type: sysbench
storage: s3://benchmark/sysbench_64_7000w
db: sysbench_64_7000w
tableNum: 64
tableSize: 70000000
threads: 64
subType: oltp_read_write
ignoreErrors: "all"
cases:
- name: "ha_pdleader_to_pdfollower_all_network_partition"
warmUpTime: 1m
faultType: network_partition
selector: "pdleader_to_pdfollower(all)"
period: "@every 5m"
faultDuration: 2m
faultTotalRunTime: 30m
ddl:
addIndex:
- db: "sysbench_64_7000w"
table: "sbtest1"
timeout: 60m
times: 5
index_col_name: "c"
job_type: "ingest"
admin_check: false
count_check: false

### 2. What did you expect to see? (Required)
add index can success

### 3. What did you see instead (Required)
add index failed with error “Error 1105 (HY000): mutex: session is expired”

add index failed at 2023-11-12 14:14:05 (Error 1105 (HY000): mutex: session is expired)
operatorLogs:
[2023-11-12 14:10:07] ###### start adding index
alter table sbtest1 add index index_test_1699769407963 (c)
[2023-11-12 14:10:07] ###### wait for ddl job finish

tidb logs:
[2023/11/12 14:12:48.454 +08:00] [ERROR] [backfilling_operators.go:751] ["flush error"] [category=ddl] [error="mutex: session is expired"]
[2023/11/12 14:12:48.454 +08:00] [ERROR] [scheduler.go:491] [onError] [task-id=180071] [error="mutex: session is expired"] [stack="[github.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*BaseScheduler).onError\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/scheduler.go:491\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*BaseScheduler).runSubtask\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/scheduler.go:290\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*BaseScheduler).run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/scheduler.go:270\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*BaseScheduler).Run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/scheduler.go:137\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTask\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/manager.go:387\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTasks.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/manager.go:217\ngithub.com/pingcap/tidb/pkg/resourcemanager/pool/spool.(*Pool).run.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/resourcemanager/pool/spool/spool.go:145](http://github.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*BaseScheduler).onError/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/scheduler.go:491/ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*BaseScheduler).runSubtask/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/scheduler.go:290/ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*BaseScheduler).run/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/scheduler.go:270/ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*BaseScheduler).Run/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/scheduler.go:137/ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTask/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/manager.go:387/ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTasks.func1/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/manager.go:217/ngithub.com/pingcap/tidb/pkg/resourcemanager/pool/spool.(*Pool).run.func1/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/resourcemanager/pool/spool/spool.go:145)"]
[2023/11/12 14:12:48.454 +08:00] [ERROR] [scheduler.go:497] ["scheduler met first error"] [task-id=180071] [error="mutex: session is expired"]
[2023/11/12 14:12:48.454 +08:00] [WARN] [scheduler.go:619] ["subtask failed"] [task-id=180071] [error="mutex: session is expired"]

### 4. What is your TiDB version? (Required)
git hash:700beafa79844b7b48dcba1c452ea3ff49d8f271

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.