add index blocked with error "not found instance” after injection pd leader io delay
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
1、enable tidb_enable_dist_task and global sort
2、run sysbench
3、add index for sbtest1 and sbtest2 concurrently
4、inject pd leader io delay 5s
operator logs:
the status of ddl job is not synced after 1h0m0s (now: 2024-05-14 21:05:56, jobId: 546, job type: add index /* ingest cloud */, state: running)
operatorLogs:
[2024-05-14 20:05:50] ###### start adding index
ALTER TABLE `sbtest2` ADD INDEX `index_test_1715688350713`(`c`)
[2024-05-14 20:05:50] ###### wait for ddl job finish
[2024-05-14 21:05:56] ###### wait for ddl job finish timeout(1h0m0s)
select job_id, job_type, state from information_schema.ddl_jobs where query = 'ALTER TABLE `sbtest2` ADD INDEX `index_test_1715688350713`(`c`)'
jobId: 546, job type: add index /* ingest cloud */, state: running
### 2. What did you expect to see? (Required)
add index can success after fault recover
### 3. What did you see instead (Required)
add index blocked with error "not found instance”
`[2024/05/14 23:31:22.997 +08:00] [INFO] [scheduler.go:264] ["schedule task meet err, reschedule it"] [task-id=78] [task-type=backfill] [allocated-slots=true] [error="not found instance"]`
### 4. What is your TiDB version? (Required)
./tidb-server -V
Release Version: v8.1.0-alpha
Edition: Community
Git Commit Hash: 48a362641a67343fddf6ddcf0f59bcaeb17be5d5
Git Branch: heads/refs/tags/v8.1.0-alpha
UTC Build Time: 2024-05-14 07:09:37
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-05-14T15:24:02.701+0800
Contributor guide
Assessment
This issue has not been assessed yet.