pingcap / pingcap/tidb

add index is always running after injecting pd leader io delay 1s last for 1mins and then recover

Open
#51,255 2 comments 0 reactions 1 assignee Claimed by @ywqzzy View on GitHub
affects-8.1 affects-8.5 component/ddl may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
1、tidb_enable_dist_task='on'
2、run sysbench
3、inject pd leader io delay 1s last for 1mins

logs:
[tidb-0.tar.gz](https://github.com/pingcap/tidb/files/14372795/tidb-0.tar.gz)
[tidb-1.tar.gz](https://github.com/pingcap/tidb/files/14372805/tidb-1.tar.gz)

### 2. What did you expect to see? (Required)
add index can success

### 3. What did you see instead (Required)
add index is always running after fault recover

the status of ddl job is not synced after 1h0m0s (now: 2024-02-21 02:27:46, jobId: 554, job type: add index /* ingest */, state: running)
operatorLogs:
[2024-02-21 01:27:20] ###### start adding index
alter table sbtest1 add index index_test_1708450040691 (c)
[2024-02-21 01:27:20] ###### wait for ddl job finish
[2024-02-21 02:27:46] ###### wait for ddl job finish timeout(1h0m0s)
select job_id, job_type, state from information_schema.ddl_jobs where query = 'alter table sbtest1 add index index_test_1708450040691 (c)'
jobId: 554, job type: add index /* ingest */, state: running"

tidb log always print:
[2024/02/21 02:26:56.039 +08:00] [INFO] [reorg.go:264] ["run reorg job wait timeout"] [category=ddl] ["wait time"=5s] ["total added row count"=70008818]
[2024/02/21 02:26:56.040 +08:00]
[INFO] [ddl_worker.go:1381] ["schema version doesn't change"] [category=ddl] [jobID=554]

### 4. What is your TiDB version? (Required)
./tidb-server -V
Release Version: v8.0.0-alpha
Edition: Community
Git Commit Hash: aca145b164905ebe48f03c46a54843234e2c99ba
Git Branch: heads/refs/tags/v8.0.0-alpha
UTC Build Time: 2024-02-18 12:22:27
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-02-21T01:05:50.694+0800

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.