sql-server: an incomplete leftover directory from an interrupted CREATE DATABASE / DOLT_CLONE wedges every retry and cannot be cleared in-process
- Dominant language
- Go
- Stars
- 24.4k
- Forks
- 873
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 108
Description
## Summary
When a `CREATE DATABASE` or `CALL DOLT_CLONE(...)` is interrupted (client crash, cancelled context, transport flap), it can leave a directory on disk that database discovery ignores but that still occupies the name. A later `CREATE`/`CLONE` of the same name then fails and **cannot recover in-process** — the only remedy is to delete the directory out of band, which races a live `dolt sql-server`.
Reproduced on **dolt 2.2.2** and present on current `main`.
## The three leftover shapes
An interrupted create/clone can leave a directory that discovery skips (logging `skipping in-progress database directory` / `skipping incomplete database directory`):
1. a directory carrying the in-progress marker `.dolt_safe_to_ignore`;
2. a directory with `.dolt` storage but no `.dolt/repo_state.json`;
3. a **bare** directory with no `.dolt` storage at all — a clone cancelled before any `.dolt` was written.
## Repro (standalone, no server required)
```bash
mkdir demo && cd demo && dolt init
# Simulate an interrupted create/clone leftover occupying the name "foo".
# Pick any one of the three shapes:
mkdir foo && touch foo/.dolt_safe_to_ignore # shape 1: in-progress marker
# mkdir -p foo/.dolt/noms # shape 2: partial .dolt, no repo_state.json
# mkdir foo # shape 3: bare dir, no .dolt
dolt sql -q "CREATE DATABASE foo;"
```
### Actual behavior
```
# shapes 1 and 2:
error on line 1 for query CREATE DATABASE foo: cannot create database foo: incomplete database directory from an interrupted create already exists; remove the directory and try again
# shape 3 (bare dir):
error on line 1 for query CREATE DATABASE foo: can't create database foo; database exists
```
`CALL DOLT_CLONE('', 'foo')` fails identically (it never reaches the remote). Meanwhile:
- `SHOW DATABASES` does **not** list `foo` (discovery ignores the leftover), and
- there is **no SQL** that clears it: `DROP DATABASE foo` and `CALL dolt_undrop('foo')` don't see it either.
The directory must be removed from the filesystem. On a running `dolt sql-server` that out-of-band `rm` races the server, and until then the name is permanently wedged.
### Expected behavior
A retried `CREATE`/`CLONE` of a name occupied **only** by an incomplete, non-servable leftover should succeed (reclaim the name) with no out-of-band filesystem surgery. A **complete, servable** database directory must of course still conflict with `database exists`.
## Impact
Any automated recovery that retries a clone after a failure — e.g. a drop → purge → reclone loop, or orchestration that reclones a replica after a transport error — gets permanently stuck: the target name is neither usable (discovery skips it) nor recreatable (`CREATE`/`CLONE` errors), and clearing it means stopping or racing the live server.
## Proposed fix
PR #11527 reclaims such a leftover at name-reservation time (under the provider lock): an incomplete, non-servable directory is moved into the `.dolt_dropped_databases` holding area (recoverable via `dolt_undrop`, purged by `dolt_purge_dropped_databases`) and the name reported available; a complete database still conflicts. It also extends `IsIncompleteDatabaseDir` to recognise the bare no-`.dolt` case (shape 3).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the standalone CREATE DATABASE reproduction for the three leftover directory shapes, then read the name-reservation path and IsIncompleteDatabaseDir described in PR #11527. Done means retrying CREATE or DOLT_CLONE reclaims incomplete leftovers without filesystem removal, while a complete servable database still reports a conflict.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100