sql-server panics fatally on shutdown when an external process deletes a database directory; recovery from resulting journal state is unreachable
- Dominant language
- Go
- Stars
- 24.4k
- Forks
- 873
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 108
Description
## Summary
When an external process deletes a per-database directory under the dolt server's data dir while the server has the database open, the server fatally panics during \`CloseAllLocalDatabases\` with:
\`\`\`
fatal error: open //.dolt/noms/nbs_manifest_NNNNNN: no such file or directory:
fatal error closing table persister
\`\`\`
After that panic, every subsequent server start emits \`possible data loss detected in journal file at offset NNNN: corrupted journal\` and fails to come up. We observed >1000 crash-loop attempts before the operator noticed and intervened. There is no built-in path to recover the journal from this state without manual surgery.
We are aware that the originating action (deleting a DB dir under a running server) is incorrect — we have a separate fix for the orchestrator that did this. This report is about dolt's failure mode being unnecessarily destructive: a missing manifest from one database during shutdown takes down the whole server and corrupts the journal for every other database it had loaded.
## Repro outline
1. Start \`dolt sql-server\` against a multi-database data_dir (e.g. \`db_a\`, \`db_b\`).
2. While the server is up, externally \`rm -rf /db_b/\`.
3. Issue any operation that exercises \`db_b\`, then send the server SIGTERM.
4. Observe panic during \`CloseAllLocalDatabases\` → \`(*GenerationalNBS).Close\` → \`(*NomsBlockStore).Close\`.
5. Restart server. Observe repeated \`possible data loss detected in journal file at offset N\` and inability to recover.
## Stack trace from our incident (dolt 1.86.6)
\`\`\`
panic: goroutine 1 [running]:
runtime/debug.Stack()
runtime/debug/stack.go:26 +0x64
github.com/dolthub/dolt/go/libraries/utils/errors.Fatalf(0xf47c08a8600?, {0x35975f4, 0x27}, {0xf48356a55f0, 0x1, 0x7959ac?})
github.com/dolthub/dolt/go/libraries/utils/errors/panic.go:39 +0x38
github.com/dolthub/dolt/go/store/nbs.(*NomsBlockStore).Close(0xf4767c7ea80)
github.com/dolthub/dolt/go/store/nbs/store.go:1673 +0xc8
github.com/dolthub/dolt/go/store/nbs.(*GenerationalNBS).Close(0xf4767af9428)
github.com/dolthub/dolt/go/store/nbs/generational_chunk_store.go:343 +0x38
github.com/dolthub/dolt/go/store/types.(*ValueStore).Close(...)
github.com/dolthub/dolt/go/store/types/value_store.go:806
github.com/dolthub/dolt/go/store/datas.(*database).Close(0x0?)
github.com/dolthub/dolt/go/store/datas/database_common.go:199 +0x28
github.com/dolthub/dolt/go/libraries/doltcore/dbfactory.CloseAllLocalDatabases()
github.com/dolthub/dolt/go/libraries/doltcore/dbfactory/file.go:97 +0x120
main.runMain()
github.com/dolthub/dolt/go/cmd/dolt/dolt.go:579 +0x2574
main.main()
github.com/dolthub/dolt/go/cmd/dolt/dolt.go:227 +0x3c
fatal error: open .../db_b/.dolt/noms/nbs_manifest_289032723:
no such file or directory: fatal error closing table persister
\`\`\`
## What would help
1. **Don't \`Fatalf\` from \`(*NomsBlockStore).Close\` for ENOENT on the manifest temp.** The DB is gone; the close has nothing left to flush. Log + skip that DB and let \`CloseAllLocalDatabases\` continue with the others. The whole-process kill on shutdown poisons every other DB's journal.
2. **Recoverable journal state for missing-DB references.** Today \`possible data loss detected\` is sticky — the operator has to manually quarantine. A documented \`fsck --revive-journal-with-data-loss\`-style escape hatch (or auto-truncation behind a flag) would unblock recovery without dropping back to a JSONL snapshot.
Related: #10837 covers a downstream symptom (\`dolt log --oneline\` hang after \`fsck --revive-journal-with-data-loss\` on a recovered repo).
## Why I'm filing this
Our orchestrator (gas-city) had a recurring corruption that we have a separate fix for. But: even with the orchestrator fixed, any operator action — manual cleanup, container churn, an unlucky \`rm\` — that disturbs a DB dir while the server is up should be a "lose that DB" event, not a "lose the entire dolt instance and require JSONL re-imports for everything" event. The current behavior is a steep cliff for what should be a recoverable misconfiguration.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.