add index failed with error "lock held by current process" after injection pd leader network partition
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
1、tidb_enable_dist_task='on'
2、run sysbench
3、 inject pd leader network partition last for 2mins
### 2. What did you expect to see? (Required)
add index can succeed
### 3. What did you see instead (Required)
add index blocked after injection pd leader network partition
```
[2025/01/07 18:12:53.968 +08:00] [WARN] [engine_mgr.go:65] ["build ingest engine failed"] ["job ID"=443] ["index ID"=41] [error="lock held by current process"]
[2025/01/07 18:12:53.968 +08:00] [ERROR] [backfilling_read_index.go:241] ["cannot register new engine"] [task-id=270028] [subtask-id=270055] [error="lock held by current process"] ["job ID"=443] ["index IDs"="[41]"]
[2025/01/07 18:12:53.971 +08:00] [INFO] [task_executor.go:451] ["run subtask failed"] [task-id=270028] [task-type=backfill] [subtaskID=270055] [step=read-index] [takeTime=4.956326ms] [error="lock held by current process"] [errorVerbose="lock held by current process\n(1) attached stack trace\n -- stack trace:\n | github.com/cockroachdb/pebble/vfs.defaultFS.Lock\n | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.0/vfs/file_lock_unix.go:50\n | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Lock\n | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.0/vfs/disk_health.go:679\n | github.com/cockroachdb/pebble.LockDirectory\n | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.0/open.go:1067\n | github.com/cockroachdb/pebble.Open\n | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.0/open.go:116\n | github.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngineDB\n | \t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:229\n | github.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngine\n | \t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:235\n | github.com/pingcap/tidb/pkg/lightning/backend/local.(*Backend).OpenEngine\n | \t/workspace/source/tidb/pkg/lightning/backend/local/local.go:816\n | github.com/pingcap/tidb/pkg/lightning/backend.EngineManager.OpenEngine\n | \t/workspace/source/tidb/pkg/lightning/backend/backend.go:266\n | github.com/pingcap/tidb/pkg/ddl/ingest.(*litBackendCtx).Register\n | \t/workspace/source/tidb/pkg/ddl/ingest/engine_mgr.go:63\n | github.com/pingcap/tidb/pkg/ddl.(*readIndexExecutor).buildLocalStorePipeline\n | \t/workspace/source/tidb/pkg/ddl/backfilling_read_index.go:239\n | github.com/pingcap/tidb/pkg/ddl.(*readIndexExecutor).RunSubtask\n | \t/workspace/source/tidb/pkg/ddl/backfilling_read_index.go:126\n | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask.func1\n | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:448\n | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask\n | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:449\n | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).Run\n | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:329\n | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*Manager).startTaskExecutor.func2\n | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/manager.go:377\n | github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).RunWithLog.func1\n | \t/workspace/source/tidb/pkg/util/wait_group_wrapper.go:181\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) lock held by current process\nError types: (1) *withstack.withStack (2) *errutil.leafError\ngithub.com/pingcap/errors.AddStack\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20240318064555-6bd07397691f/errors.go:178\ngithub.com/pingcap/errors.Trace\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20240318064555-6bd07397691f/juju_adaptor.go:15\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngineDB\n\t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:230\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngine\n\t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:235\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*Backend).OpenEngine\n\t/workspace/source/tidb/pkg/lightning/backend/local/local.go:816\ngithub.com/pingcap/tidb/pkg/lightning/backend.EngineManager.OpenEngine\n\t/workspace/source/tidb/pkg/lightning/backend/backend.go:266\ngithub.com/pingcap/tidb/pkg/ddl/ingest.(*litBackendCtx).Register\n\t/workspace/source/tidb/pkg/ddl/ingest/engine_mgr.go:63\ngithub.com/pingcap/tidb/pkg/ddl.(*readIndexExecutor).buildLocalStorePipeline\n\t/workspace/source/tidb/pkg/ddl/backfilling_read_index.go:239\ngithub.com/pingcap/tidb/pkg/ddl.(*readIndexExecutor).RunSubtask\n\t/workspace/source/tidb/pkg/ddl/backfilling_read_index.go:126\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask.func1\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:448\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:449\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).Run\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:329\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*Manager).startTaskExecutor.func2\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/manager.go:377\ngithub.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).RunWithLog.func1\n\t/workspace/source/tidb/pkg/util/wait_group_wrapper.go:181\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"]
[2025/01/07 18:12:53.971 +08:00] [WARN] [task_executor.go:571] ["meet retryable error"] [task-id=270028] [task-type=backfill] [error="lock held by current process"] [errorVerbose="lock held by current process\n(1) attached stack trace\n -- stack trace:\n | github.com/cockroachdb/pebble/vfs.defaultFS.Lock\n | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.0/vfs/file_lock_unix.go:50\n | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Lock\n | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.0/vfs/disk_health.go:679\n | github.com/cockroachdb/pebble.LockDirectory\n | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.0/open.go:1067\n | github.com/cockroachdb/pebble.Open\n | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.0/open.go:116\n | github.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngineDB\n | \t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:229\n | github.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngine\n | \t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:235\n | github.com/pingcap/tidb/pkg/lightning/backend/local.(*Backend).OpenEngine\n | \t/workspace/source/tidb/pkg/lightning/backend/local/local.go:816\n | github.com/pingcap/tidb/pkg/lightning/backend.EngineManager.OpenEngine\n | \t/workspace/source/tidb/pkg/lightning/backend/backend.go:266\n | github.com/pingcap/tidb/pkg/ddl/ingest.(*litBackendCtx).Register\n | \t/workspace/source/tidb/pkg/ddl/ingest/engine_mgr.go:63\n | github.com/pingcap/tidb/pkg/ddl.(*readIndexExecutor).buildLocalStorePipeline\n | \t/workspace/source/tidb/pkg/ddl/backfilling_read_index.go:239\n | github.com/pingcap/tidb/pkg/ddl.(*readIndexExecutor).RunSubtask\n | \t/workspace/source/tidb/pkg/ddl/backfilling_read_index.go:126\n | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask.func1\n | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:448\n | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask\n | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:449\n | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).Run\n | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:329\n | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*Manager).startTaskExecutor.func2\n | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/manager.go:377\n | github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).RunWithLog.func1\n | \t/workspace/source/tidb/pkg/util/wait_group_wrapper.go:181\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) lock held by current process\nError types: (1) *withstack.withStack (2) *errutil.leafError\ngithub.com/pingcap/errors.AddStack\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20240318064555-6bd07397691f/errors.go:178\ngithub.com/pingcap/errors.Trace\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20240318064555-6bd07397691f/juju_adaptor.go:15\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngineDB\n\t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:230\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngine\n\t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:235\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*Backend).OpenEngine\n\t/workspace/source/tidb/pkg/lightning/backend/local/local.go:816\ngithub.com/pingcap/tidb/pkg/lightning/backend.EngineManager.OpenEngine\n\t/workspace/source/tidb/pkg/lightning/backend/backend.go:266\ngithub.com/pingcap/tidb/pkg/ddl/ingest.(*litBackendCtx).Register\n\t/workspace/source/tidb/pkg/ddl/ingest/engine_mgr.go:63\ngithub.com/pingcap/tidb/pkg/ddl.(*readIndexExecutor).buildLocalStorePipeline\n\t/workspace/source/tidb/pkg/ddl/backfilling_read_index.go:239\ngithub.com/pingcap/tidb/pkg/ddl.(*readIndexExecutor).RunSubtask\n\t/workspace/source/tidb/pkg/ddl/backfilling_read_index.go:126\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask.func1\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:448\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:449\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).Run\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:329\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*Manager).startTaskExecutor.func2\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/manager.go:377\ngithub.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).RunWithLog.func1\n\t/workspace/source/tidb/pkg/util/wait_group_wrapper.go:181\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"]
[2025/01/07 18:12:53.971 +08:00] [ERROR] [task_executor.go:334] ["run subtask failed"] [task-id=270028] [task-type=backfill] [error="lock held by current process"] [errorVerbose="lock held by current process\n(1) attached stack trace\n -- stack trace:\n | github.com/cockroachdb/pebble/vfs.defaultFS.Lock\n | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.0/vfs/file_lock_unix.go:50\n | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Lock\n | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.0/vfs/disk_health.go:679\n | github.com/cockroachdb/pebble.LockDirectory\n | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.0/open.go:1067\n | github.com/cockroachdb/pebble.Open\n | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.0/open.go:116\n | github.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngineDB\n | \t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:229\n | github.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngine\n | \t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:235\n | github.com/pingcap/tidb/pkg/lightning/backend/local.(*Backend).OpenEngine\n | \t/workspace/source/tidb/pkg/lightning/backend/local/local.go:816\n | github.com/pingcap/tidb/pkg/lightning/backend.EngineManager.OpenEngine\n | \t/workspace/source/tidb/pkg/lightning/backend/backend.go:266\n | github.com/pingcap/tidb/pkg/ddl/ingest.(*litBackendCtx).Register\n | \t/workspace/source/tidb/pkg/ddl/ingest/engine_mgr.go:63\n | github.com/pingcap/tidb/pkg/ddl.(*readIndexExecutor).buildLocalStorePipeline\n | \t/workspace/source/tidb/pkg/ddl/backfilling_read_index.go:239\n | github.com/pingcap/tidb/pkg/ddl.(*readIndexExecutor).RunSubtask\n | \t/workspace/source/tidb/pkg/ddl/backfilling_read_index.go:126\n | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask.func1\n | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:448\n | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask\n | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:449\n | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).Run\n | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:329\n | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*Manager).startTaskExecutor.func2\n | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/manager.go:377\n | github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).RunWithLog.func1\n | \t/workspace/source/tidb/pkg/util/wait_group_wrapper.go:181\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) lock held by current process\nError types: (1) *withstack.withStack (2) *errutil.leafError\ngithub.com/pingcap/errors.AddStack\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20240318064555-6bd07397691f/errors.go:178\ngithub.com/pingcap/errors.Trace\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20240318064555-6bd07397691f/juju_adaptor.go:15\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngineDB\n\t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:230\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngine\n\t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:235\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*Backend).OpenEngine\n\t/workspace/source/tidb/pkg/lightning/backend/local/local.go:816\ngithub.com/pingcap/tidb/pkg/lightning/backend.EngineManager.OpenEngine\n\t/workspace/source/tidb/pkg/lightning/backend/backend.go:266\ngithub.com/pingcap/tidb/pkg/ddl/ingest.(*litBackendCtx).Register\n\t/workspace/source/tidb/pkg/ddl/ingest/engine_mgr.go:63\ngithub.com/pingcap/tidb/pkg/ddl.(*readIndexExecutor).buildLocalStorePipeline\n\t/workspace/source/tidb/pkg/ddl/backfilling_read_index.go:239\ngithub.com/pingcap/tidb/pkg/ddl.(*readIndexExecutor).RunSubtask\n\t/workspace/source/tidb/pkg/ddl/backfilling_read_index.go:126\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask.func1\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:448\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:449\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).Run\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:329\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*Manager).startTaskExecutor.func2\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/manager.go:377\ngithub.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).RunWithLog.func1\n\t/workspace/source/tidb/pkg/util/wait_group_wrapper.go:181\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"]
```
### 4. What is your TiDB version? (Required)
./tidb-server -V
Release Version: v9.0.0-alpha-59-gc199ddf
Edition: Community
Git Commit Hash: c199ddfcdf9e1f88a243c2b7847d8f3224446004
Git Branch: HEAD
UTC Build Time: 2025-01-07 05:37:06
GoVersion: go1.23.4
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2025-01-07T17:44:53.845+0800
Contributor guide
Assessment
This issue has not been assessed yet.