add index failed with error [Lightning:Config:ErrInvalidSortedKVDir]invalid sorted-kv-dir '/var/lib/tidb-data/tmp_ddl-4000/687' for local backend, please change the config or delete the path: mkdir /var/lib/tidb-data/tmp_ddl-4000/687: no such file or directory"] when injection tidb temp-dir io delay 1s
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
1、tidb_enable_dist_task='on'
2、run sysbench
3、add index
4、injection tidb temp-dir io delay 1s lasts for 3mins
tidb config
```
tidb:
baseImage: "{{meta.parameters.tidb_base_image}}"
config: |
temp-dir = "/var/lib/tidb-data"
limits:
cpu: 8000m
memory: 16Gi
maxReplicas: 2
replicas: 2
requests:
cpu: 8000m
memory: 16Gi
storageVolumes:
- mountPath: /var/lib/tidb-data
name: data
storageClassName: fast-disks
storageSize: 1400Gi
```
### 2. What did you expect to see? (Required)
add index can succeed
### 3. What did you see instead (Required)
add index failed
`[2024/12/20 15:48:30.647 +08:00] [ERROR] [backend_mgr.go:95] ["build ingest backend failed"] ["job ID"=687] [error="[Lightning:Config:ErrInvalidSortedKVDir]invalid sorted-kv-dir '/var/lib/tidb-data/tmp_ddl-4000/687' for local backend, please change the config or delete the path: mkdir /var/lib/tidb-data/tmp_ddl-4000/687: no such file or directory"]
[2024/12/20 15:48:30.647 +08:00] [ERROR] [manager.go:430] ["task manager met error"] [error="[Lightning:Config:ErrInvalidSortedKVDir]invalid sorted-kv-dir '/var/lib/tidb-data/tmp_ddl-4000/687' for local backend, please change the config or delete the path: mkdir /var/lib/tidb-data/tmp_ddl-4000/687: no such file or directory"] [stack="[github.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).logErr\n\t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:430\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).logErrAndPersist\n\t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:434\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTask\n\t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:345\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTasks.func1\n\t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:220\ngithub.com/pingcap/tidb/pkg/resourcemanager/pool/spool.(*Pool).run.func1\n\t/workspace/source/tidb/pkg/resourcemanager/pool/spool/spool.go:145](http://github.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).logErr/n/t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:430/ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).logErrAndPersist/n/t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:434/ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTask/n/t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:345/ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTasks.func1/n/t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:220/ngithub.com/pingcap/tidb/pkg/resourcemanager/pool/spool.(*Pool).run.func1/n/t/workspace/source/tidb/pkg/resourcemanager/pool/spool/spool.go:145)"]
[2024/12/20 15:48:30.660 +08:00] [ERROR] [manager.go:443] ["update error to subtask"] [task-id=390023] [] [stack="[github.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).logErrAndPersist\n\t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:443\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTask\n\t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:345\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTasks.func1\n\t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:220\ngithub.com/pingcap/tidb/pkg/resourcemanager/pool/spool.(*Pool).run.func1\n\t/workspace/source/tidb/pkg/resourcemanager/pool/spool/spool.go:145](http://github.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).logErrAndPersist/n/t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:443/ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTask/n/t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:345/ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTasks.func1/n/t/workspace/source/tidb/pkg/disttask/framework/scheduler/manager.go:220/ngithub.com/pingcap/tidb/pkg/resourcemanager/pool/spool.(*Pool).run.func1/n/t/workspace/source/tidb/pkg/resourcemanager/pool/spool/spool.go:145)"]
[2024/12/20 15:48:31.043 +08:00] [INFO] [manager.go:216] ["detect new subtask"] [task-id=390023]`
### 4. What is your TiDB version? (Required)
./tidb-server -V
Release Version: v7.5.5
Edition: Community
Git Commit Hash: a0a8adb1e31bc1ff6fbb62a5bc4900e344877cd7
Git Branch: HEAD
UTC Build Time: 2024-12-19 07:47:15
GoVersion: go1.21.13
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Contributor guide
Assessment
This issue has not been assessed yet.