pingcap / pingcap/tidb

[tidb-x] add index task was always blocked after simulated kill tikv-worker repeatedly

Open
#64,656 2 comments 0 reactions 1 assignee Claimed by @D3Hunter View on GitHub
component/ddl component/DXF may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
1、run sysbench
2、add index for two tables
3、simulated kill all tikv-worker repeatedly

### 2. What did you expect to see? (Required)
add index can succeed after fault recovery

### 3. What did you see instead (Required)
add index task was blocking after simulated kill tikv-worker repeatedly

mysql> select job_id, db_name, table_name, job_type, create_time, start_time, end_time, state, query from information_schema.ddl_jobs where query = 'ALTER TABLE `sbtest1` ADD INDEX `index_test_1763990711836`(`c`)';
+--------+-------------------------------+------------+-----------+----------------------------+----------------------------+----------------------------+---------------+-----------------------------------------------------------------+
| job_id | db_name | table_name | job_type | create_time | start_time | end_time | state | query |
+--------+-------------------------------+------------+-----------+----------------------------+----------------------------+----------------------------+---------------+-----------------------------------------------------------------+
| 7402 | partitioninsert_1647963896275 | sbtest1 | add index | 2025-11-24 13:25:12.127000 | 2025-11-24 13:25:12.127000 | 2025-11-25 01:28:03.027000 | rollback done | ALTER TABLE `sbtest1` ADD INDEX `index_test_1763990711836`(`c`) |
+--------+-------------------------------+------------+-----------+----------------------------+----------------------------+----------------------------+---------------+-----------------------------------------------------------------+
1 row in set (0.07 sec)

task was cancelled by user at 2025-11-25 01:28:03.027000, so the status is rollback done now

tidb log
[tidb (2).log.zip](https://github.com/user-attachments/files/23736294/tidb.2.log.zip)

tidb-worker log

[tidb-worker.log.zip](https://github.com/user-attachments/files/23739827/tidb-worker.log.zip)

[2025/11/24 20:58:27.846 +00:00] [WARN] [job_worker.go:218] ["meet retryable error when ingesting, will handle the job later"] [keyspaceName=SYSTEM] [task-id=3] [task-key=keyspace_a/ddl/backfill/7402] [subtaskID=18] [step=ingest] [error="[Ingest:ErrKVIngestFailed]ingest files 462426362197311506 overlap with existing files, can't ingest to level 0(ingest SST ID 462426362197311506)"] ["job stage"=regionScanned] [region="{ID=1818,startKey=7800000174800000FF0000000F2A5F6980FF0000000000000301FF3633333132393430FFFF3732392D303136FF35FF373034373531FF342DFF3935343035FF363536FF3533322DFF30363733FF353638FF393037362DFF3933FF323231333733FF37FF36352D31343135FFFF323138373639312DFFFF37393537393136FF30FF3737392D3030FF3935FF3635333230FF39302DFF30393235FF35343337FF343135FF2D30363338FF3332FF373437393700FE03FF80000000000C8F4AFF0000000000000000F7,endKey=7800000174800000FF0000000F2A5F72FFFFFFFFFFFFFFFFFF00FF0000000000000000F7,epoch=\"conf_ver:5 version:199 \",peers=\"id:1819 store_id:1001 ,id:1820 store_id:1276 ,id:1821 store_id:1341 \"}"]

### 4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v9.0.0-beta.2.pre-810-gbf57e58
Edition: Enterprise
Git Commit Hash: bf57e58556e73b679c7c774a05392694eca8e25a
Git Branch: HEAD
UTC Build Time: 2025-11-24 03:31:04
GoVersion: go1.23.12
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Enterprise Extension Commit Hash: dc633aae52eb11b4e3549e3a7ef5b0ed14e159b1
Kernel Type: Next Generation

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.