pingcap / pingcap/tidb

[tidb-x] add index failed with error="[tikv:9001]PD server timeout: start timestamp may fall behind safe point"] when simulated pd network latency

Open
#65,333 1 comment 0 reactions 1 assignee Claimed by @GMHDBJD View on GitHub
component/DXF severity/moderate type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
1、enable global sort and dxf
2、run sysbench
3、add index for one table
4、simulated pd network latency

case logs:
Test case ######ha_pdleader_to_all_network_latency(latency:50ms)###### failed
the state of ddl job is not synced (now: 2025-12-29 23:14:18, jobId: 381, job type: add index, state: rollback done, comments: )
operatorLogs:
[2025-12-29 22:58:17] ###### start adding index
ALTER TABLE `sbtest1` ADD INDEX `index_test_1767020297135`(`c`)
[2025-12-29 22:58:17] ###### wait for ddl job finish
[2025-12-29 23:14:18] ###### ddl job failed
select job_id, db_name, table_name, job_type, create_time, start_time, end_time, state, query from information_schema.ddl_jobs where query = 'ALTER TABLE `sbtest1` ADD INDEX `index_test_1767020297135`(`c`)'
jobId: 381, job type: add index, state: rollback done, comments:

### 2. What did you expect to see? (Required)
add index can succeed

### 3. What did you see instead (Required)
add index failed with error="[tikv:9001]PD server timeout: start timestamp may fall behind safe point"]

`2025/12/29 23:14:16.363 +08:00] [ERROR] [handle.go:143] ["task reverted"] [keyspaceName=keyspace_a] [task-id=18] [error="[tikv:9001]PD server timeout: start timestamp may fall behind safe point"]
[2025/12/29 23:14:16.371 +08:00] [WARN] [reorg.go:431] ["run reorg job done"] [keyspaceName=keyspace_a] [category=ddl] [jobID=381] ["handled rows"=0] [error="[tikv:9001]PD server timeout: start timestamp may fall behind safe point"]
[2025/12/29 23:14:16.371 +08:00] [WARN] [index.go:1970] ["run add index job failed, convert job to rollback"] [keyspaceName=keyspace_a] [category=ddl] [job="ID:381, Type:add index, State:running, SchemaState:write reorganization, SchemaID:357, TableID:245, RowCount:0, ArgLen:1, start time: 2025-12-29 22:58:17.308 +0800 CST, Err:[tikv:9001]PD server timeout: start timestamp may fall behind safe point, ErrCount:511, SnapshotVersion:463213769577988308, Version: v2, UniqueWarnings:0"] [error="[tikv:9001]PD server timeout: start timestamp may fall behind safe point"]`

### 4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v9.0.0-beta.2.pre-975-g07f3e75
Edition: Enterprise
Git Commit Hash: 07f3e7585a254d9c7b689822421f91d33922def4
Git Branch: HEAD
UTC Build Time: 2025-12-29 09:35:50
GoVersion: go1.25.5
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Enterprise Extension Commit Hash: dc633aae52eb11b4e3549e3a7ef5b0ed14e159b1
Kernel Type: Next Generation
2025-12-29T22:38:27.181+0800

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.