add index is always running after injecting pd leader io delay 1s last for 1mins and then recover
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
1、tidb_enable_dist_task='on'
2、run sysbench
3、inject pd leader io delay 1s last for 1mins
logs:
[tidb-0.tar.gz](https://github.com/pingcap/tidb/files/14372795/tidb-0.tar.gz)
[tidb-1.tar.gz](https://github.com/pingcap/tidb/files/14372805/tidb-1.tar.gz)
### 2. What did you expect to see? (Required)
add index can success
### 3. What did you see instead (Required)
add index is always running after fault recover
the status of ddl job is not synced after 1h0m0s (now: 2024-02-21 02:27:46, jobId: 554, job type: add index /* ingest */, state: running)
operatorLogs:
[2024-02-21 01:27:20] ###### start adding index
alter table sbtest1 add index index_test_1708450040691 (c)
[2024-02-21 01:27:20] ###### wait for ddl job finish
[2024-02-21 02:27:46] ###### wait for ddl job finish timeout(1h0m0s)
select job_id, job_type, state from information_schema.ddl_jobs where query = 'alter table sbtest1 add index index_test_1708450040691 (c)'
jobId: 554, job type: add index /* ingest */, state: running"
tidb log always print:
[2024/02/21 02:26:56.039 +08:00] [INFO] [reorg.go:264] ["run reorg job wait timeout"] [category=ddl] ["wait time"=5s] ["total added row count"=70008818]
[2024/02/21 02:26:56.040 +08:00]
[INFO] [ddl_worker.go:1381] ["schema version doesn't change"] [category=ddl] [jobID=554]
### 4. What is your TiDB version? (Required)
./tidb-server -V
Release Version: v8.0.0-alpha
Edition: Community
Git Commit Hash: aca145b164905ebe48f03c46a54843234e2c99ba
Git Branch: heads/refs/tags/v8.0.0-alpha
UTC Build Time: 2024-02-18 12:22:27
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-02-21T01:05:50.694+0800
Contributor guide
Assessment
This issue has not been assessed yet.