pingcap / pingcap/tidb

add index failed with error [Lightning:Config:ErrInvalidSortedKVDir]invalid sorted-kv-dir '/var/lib/tidb-data/tmp_ddl-4000/687' for local backend, please change the config or delete the path: mkdir /var/lib/tidb-data/tmp_ddl-4000/687: no such file or directory"] when injection tidb temp-dir io delay 1s

Open
#58,437 0 comments 0 reactions 0 assignees View on GitHub
component/ddl may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
1、tidb_enable_dist_task='on'
2、run sysbench
3、add index
4、injection tidb temp-dir io delay 1s lasts for 3mins

tidb config
```
tidb:
baseImage: "{{meta.parameters.tidb_base_image}}"
config: |
temp-dir = "/var/lib/tidb-data"
limits:
cpu: 8000m
memory: 16Gi
maxReplicas: 2
replicas: 2
requests:
cpu: 8000m
memory: 16Gi
storageVolumes:
- mountPath: /var/lib/tidb-data
name: data
storageClassName: fast-disks
storageSize: 1400Gi
```

### 2. What did you expect to see? (Required)
add index can succeed

### 3. What did you see instead (Required)
add index failed
`[2024/12/20 15:48:30.647 +08:00] [ERROR] [backend_mgr.go:95] ["build ingest backend failed"] ["job ID"=687] [error="[Lightning:Config:ErrInvalidSortedKVDir]invalid sorted-kv-dir '/var/lib/tidb-data/tmp_ddl-4000/687' for local backend, please change the config or delete the path: mkdir /var/lib/tidb-data/tmp_ddl-4000/687: no such file or directory"]
[2024/12/20 15:48:30.647 +08:00] [ERROR] [manager.go:430] ["task manager met error"] [error="[Lightning:Config:ErrInvalidSortedKVDir]invalid sorted-kv-dir '/var/lib/tidb-data/tmp_ddl-4000/687' for local backend, please change the config or delete the path: mkdir /var/lib/tidb-data/tmp_ddl-4000/687: no such file or directory"] [stack="[github.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).logErr\n\t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:430\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).logErrAndPersist\n\t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:434\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTask\n\t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:345\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTasks.func1\n\t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:220\ngithub.com/pingcap/tidb/pkg/resourcemanager/pool/spool.(*Pool).run.func1\n\t/workspace/source/tidb/pkg/resourcemanager/pool/spool/spool.go:145](http://github.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).logErr/n/t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:430/ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).logErrAndPersist/n/t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:434/ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTask/n/t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:345/ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTasks.func1/n/t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:220/ngithub.com/pingcap/tidb/pkg/resourcemanager/pool/spool.(*Pool).run.func1/n/t/workspace/source/tidb/pkg/resourcemanager/pool/spool/spool.go:145)"]
[2024/12/20 15:48:30.660 +08:00] [ERROR] [manager.go:443] ["update error to subtask"] [task-id=390023] [] [stack="[github.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).logErrAndPersist\n\t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:443\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTask\n\t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:345\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTasks.func1\n\t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:220\ngithub.com/pingcap/tidb/pkg/resourcemanager/pool/spool.(*Pool).run.func1\n\t/workspace/source/tidb/pkg/resourcemanager/pool/spool/spool.go:145](http://github.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).logErrAndPersist/n/t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:443/ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTask/n/t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:345/ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTasks.func1/n/t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:220/ngithub.com/pingcap/tidb/pkg/resourcemanager/pool/spool.(*Pool).run.func1/n/t/workspace/source/tidb/pkg/resourcemanager/pool/spool/spool.go:145)"]
[2024/12/20 15:48:31.043 +08:00] [INFO] [manager.go:216] ["detect new subtask"] [task-id=390023]`

### 4. What is your TiDB version? (Required)
./tidb-server -V
Release Version: v7.5.5
Edition: Community
Git Commit Hash: a0a8adb1e31bc1ff6fbb62a5bc4900e344877cd7
Git Branch: HEAD
UTC Build Time: 2024-12-19 07:47:15
GoVersion: go1.21.13
Race Enabled: false
Check Table Before Drop: false
Store: unistore

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.