ClickHouse / ClickHouse/ClickHouse
`EmbeddedRocksDB` with a custom `rocksdb_dir`: changing `user_files_path` silently empties relative-path tables and makes absolute-path tables undroppable
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
**TL;DR:** After `user_files_path` is changed in the server config, an `EmbeddedRocksDB` table with a custom relative `rocksdb_dir` silently comes up **empty** (the data stays in the old directory, a new empty RocksDB is created at the new resolution), and a table with a custom absolute `rocksdb_dir` under the old `user_files_path` cannot be dropped or detached and breaks `system.tables`; with `async_load_databases = false` the server refuses to start.
Sibling of #119702 (`File` engine, same `user_files_path`-change trigger) and of #111958 (same silent-empty consequence, `RENAME` trigger).
1. **Relative `rocksdb_dir` → silent data loss.** A custom relative dir is resolved against `user_files_path` at every construction (src/Storages/RocksDB/StorageEmbeddedRocksDB.cpp:262). After the config change the table opens (and creates) a brand-new empty RocksDB at `/` — `SELECT count()` returns 0 with no error, and the real data is orphaned at `/`. Unlike the `File` engine, where relative paths re-resolve to the same user-visible files, here the storage owns the data, so re-resolution detaches the table from it.
2. **Absolute `rocksdb_dir` → bricked table.** The containment check in the constructor (src/Storages/RocksDB/StorageEmbeddedRocksDB.cpp:265) has no `LoadingStrictnessLevel` gate, so it re-fails during metadata replay. With default async loading, `DROP TABLE`, `DETACH TABLE` and every query touching `system.tables` throw `ASYNC_LOAD_WAIT_FAILED` wrapping `BAD_ARGUMENTS` — there is no SQL escape. With `async_load_databases = false` the server exits during metadata load (`Caught exception while loading metadata`). #87557 established that enforcing the fence for pre-fence tables is intended, but these tables were created fully inside the fence and only the config moved; even if refusing to read them is intended, `DROP`/`DETACH` should still work.
**Does it reproduce on the most recent release?**
Reproduced on master 26.9.1.1307 and on 26.9.1.1.
**How to reproduce**
With `/data/user_files_a/`:
```sql
CREATE TABLE rocks_abs (k UInt64, v String) ENGINE = EmbeddedRocksDB(0, '/data/user_files_a/rocks_abs') PRIMARY KEY k;
CREATE TABLE rocks_rel (k UInt64, v String) ENGINE = EmbeddedRocksDB(0, 'rocks_rel') PRIMARY KEY k;
INSERT INTO rocks_abs SELECT number, toString(number) FROM numbers(500);
INSERT INTO rocks_rel SELECT number, toString(number) FROM numbers(500);
```
Stop the server cleanly, change `user_files_path` to `/data/user_files_b/`, restart:
```sql
SELECT count() FROM rocks_rel; -- 0, silently; the 500 rows sit in user_files_a/rocks_rel, and an empty DB now exists in user_files_b/rocks_rel
DROP TABLE rocks_abs; -- ASYNC_LOAD_WAIT_FAILED wrapping BAD_ARGUMENTS "Path must be inside user-files path"
SELECT name FROM system.tables; -- same exception
```
Control: reverting `user_files_path` restores both tables with all 500 rows each.
**Expected behavior**
The relative-path table keeps its data (resolve the custom dir once at creation and persist it, or refuse to open a different directory than the one that holds the data instead of silently creating an empty one). The absolute-path table remains at least droppable: the containment check should be gated on `LoadingStrictnessLevel` during metadata replay, as `StorageFileLog` already does for the same check (src/Storages/FileLog/StorageFileLog.cpp:193).
**Error message and/or stacktrace**
```
Code: 722. DB::Exception: Waited job failed: Code: 696. DB::Exception: Load job 'startup table default.rocks_abs' -> Code: 695.
DB::Exception: Load job 'load table default.rocks_abs' failed: Code: 36. DB::Exception: Path must be inside user-files path:
/data/user_files_b: Cannot attach table `default`.`rocks_abs` ... ENGINE = EmbeddedRocksDB(0, '/data/user_files_a/rocks_abs')
PRIMARY KEY k. (BAD_ARGUMENTS),. (ASYNC_LOAD_WAIT_FAILED)
```
Contributor guide
Assessment
This issue has not been assessed yet.