pingcap / pingcap/tidb

add index blocked with error "not found instance” after injection pd leader io delay

Open
#53,282 3 comments 0 reactions 1 assignee Claimed by @D3Hunter View on GitHub
affects-8.5 component/ddl may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
1、enable tidb_enable_dist_task and global sort
2、run sysbench
3、add index for sbtest1 and sbtest2 concurrently
4、inject pd leader io delay 5s

operator logs:
the status of ddl job is not synced after 1h0m0s (now: 2024-05-14 21:05:56, jobId: 546, job type: add index /* ingest cloud */, state: running)
operatorLogs:
[2024-05-14 20:05:50] ###### start adding index
ALTER TABLE `sbtest2` ADD INDEX `index_test_1715688350713`(`c`)
[2024-05-14 20:05:50] ###### wait for ddl job finish
[2024-05-14 21:05:56] ###### wait for ddl job finish timeout(1h0m0s)
select job_id, job_type, state from information_schema.ddl_jobs where query = 'ALTER TABLE `sbtest2` ADD INDEX `index_test_1715688350713`(`c`)'
jobId: 546, job type: add index /* ingest cloud */, state: running

### 2. What did you expect to see? (Required)
add index can success after fault recover

### 3. What did you see instead (Required)
add index blocked with error "not found instance”
`[2024/05/14 23:31:22.997 +08:00] [INFO] [scheduler.go:264] ["schedule task meet err, reschedule it"] [task-id=78] [task-type=backfill] [allocated-slots=true] [error="not found instance"]`

### 4. What is your TiDB version? (Required)
./tidb-server -V
Release Version: v8.1.0-alpha
Edition: Community
Git Commit Hash: 48a362641a67343fddf6ddcf0f59bcaeb17be5d5
Git Branch: heads/refs/tags/v8.1.0-alpha
UTC Build Time: 2024-05-14 07:09:37
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-05-14T15:24:02.701+0800

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.