dolthub / dolthub/dolt

Dolt Fails to Start After Restart in Kubernetes STS with ProxySQL – Reports Database Locked by Another Process

Open
#11,036 3 comments 0 reactions 0 assignees View on GitHub
bug cli concurrency customer issue known workaround no repro question sql server
Dominant language
Go
Stars
24.4k
Forks
873
Avg merge
1d 5h
Merged PRs (30d)
108

Description

We are deploying Dolt using Kubernetes StatefulSet (STS). The Dolt version is 2.0.1.

```
root@dolt-0:/db/dolt# dolt version
Warning: Unknown global config option 'sqlserver.cluster.dolt_cluster_role'. Use `dolt config --global --unset sqlserver.cluster.dolt_cluster_role` to remove.
Warning: Unknown global config option 'sqlserver.cluster.dolt_cluster_role_epoch'. Use `dolt config --global --unset sqlserver.cluster.dolt_cluster_role_epoch` to remove.
dolt version 2.0.1
```

The application does not connect directly to Dolt, but connects via ProxySQL. ProxySQL version is 3.0.7.

When Dolt restarts, it fails to start. The log is as follows:

```
Starting server with Config HP="0.0.0.0:6033"|T="28800000"|R="false"|L="info"
database "insight_jobadmin" is locked by another dolt process; either clone the database to run a second server, or stop the dolt process which currently holds an exclusive write lock on the database
PS C:\Users\LSX17> ksit logs -f gitee-insight-v3-miji-insight-dolt-0
Defaulted container "insight-dolt" out of: insight-dolt, init-config (init)
Starting periodic command scheduler: cron.
Starting server with Config HP="0.0.0.0:6033"|T="28800000"|R="false"|L="info"
database "insight_datawarehouse" is locked by another dolt process; either clone the database to run a second server, or stop the dolt process which currently holds an exclusive write lock on the database
```

As shown in the logs above, the database name "insight_datawarehouse" is not fixed. Each time it restarts, a new database name appears here, but the rest of the error message is identical.

```
ls -alh /db/dolt/insight_datawarehouse/.dolt
total 28K
drwxr-xr-x 5 root root 4.0K May 12 10:46 .
drwxr-xr-x 3 root root 4.0K May 12 10:41 ..
-rwxrwxrwx 1 root root 2 May 12 10:41 config.json
drwxr-xr-x 3 root root 4.0K May 12 14:08 noms
-rwxrwxrwx 1 root root 387 May 12 10:46 repo_state.json
drwxr-xr-x 3 root root 4.0K May 12 10:41 stats
drwxr-xr-x 2 root root 4.0K May 12 10:47 temptf
```

What is the cause of this problem? How can it be resolved? I have redeployed the cluster multiple times, and every time Dolt restarts, this issue occurs.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.