ClickHouse / ClickHouse/ClickHouse
`system.webassembly_modules`: acknowledged module INSERT is not durable (no fsync, in-place O_TRUNC) — power loss loses the module or bricks server startup (Code 117)
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Describe what's wrong
Uploading a WebAssembly module with `INSERT INTO system.webassembly_modules` returns success **before the module file on disk is durable**. The module is stored as `user_scripts/wasm/.wasm`, and that file is written in place (truncate + write) and is never `fsync`ed — neither the file nor its directory.
By contrast, `CREATE FUNCTION ... LANGUAGE WASM` persists the function it creates **durably** (it writes a temporary `.sql`, `fdatasync`s it, then atomically renames it into place). So once both statements are acknowledged, we have a durable function definition pointing at a module file that is not durable.
If the server loses power (or the machine crashes) after those acknowledged statements, two things can happen:
**1. The uploaded module is silently lost.** After the crash the `.wasm` file is empty or truncated, and startup treats a too-short file as "not a valid module" and ignores it. The row disappears from `system.webassembly_modules` even though the `INSERT` was acknowledged.
**2. The server refuses to start at all.** If a `CREATE FUNCTION ... LANGUAGE WASM` referenced that module, the function definition survives the crash but the module does not. On startup ClickHouse loads user-defined functions, fails to load the WASM module, and the exception aborts server startup:
```
Code: 117. DB::Exception: Cannot load WebAssembly module 'm1':
File 'wasm/m1.wasm' is too small to be a valid WebAssembly module:
while loading user defined function `isp`. (INCORRECT_DATA)
```
The server then stays down until an administrator manually deletes the orphaned function `.sql` or the broken module file.
Two related problems in the same code:
- Because the module is written with truncation (not written to a temp file and renamed), a torn write can leave a partially-written module that keeps its 8-byte magic header. Such a module registers on startup but fails to compile when called, and re-uploading the correct bytes then fails with `FILE_ALREADY_EXISTS` — so it cannot be repaired through SQL.
- `deleteModuleIfExists` removes the module file without an `fsync` of the directory, so an acknowledged module deletion can also be undone by a crash.
### Does it reproduce on the most recent release?
Yes, reproduces on `master` (26.7.1.1380). The feature was introduced in v26.3.1, so every release since is affected. Requires the server setting `allow_experimental_webassembly_udf`.
### How to reproduce
Both statements below return success:
```sql
INSERT INTO system.webassembly_modules (name, code, hash)
SELECT
'm1',
base64Decode('AGFzbQEAAAABBgFgAX8BfwMCAQAHDAEIaXNfcHJpbWUAAApUAVIBAX8gAEECSQRAQQAPCyAAQQJGBEBBAQ8LIABBAnBBAEYEQEEADwtBAyEBA0ACQCABIABPDQAgACABcEEARgRAQQAPCyABQQJqIQEMAQsLQQEL'),
reinterpretAsUInt256(unhex('369f6098ed1ac69a97735c26039f090ab78c31d729e8110f086b2ea13611c57d'));
CREATE FUNCTION isp LANGUAGE WASM ABI ROW_DIRECT
FROM 'm1' :: 'is_prime' ARGUMENTS (num UInt32) RETURNS UInt32;
-- kill power to the machine here
```
After the power loss, restart the server: it fails to start with the `Code: 117` error shown above.
I reproduced this with a faithful power-loss simulation: the data directory on an `ext4`/`xfs` `dm-flakey` device, switched to `drop_writes`, then unmounted and remounted so only `fsync`ed data survives — the exact state a power cut leaves. It reproduces 3/3 on ext4 and on xfs. A control file that the test `fsync`s itself survives the same cut, confirming the loss is ClickHouse's and not the test's.
strace showing the asymmetry (no crash required)
The module `INSERT` writes the file in place and never `fsync`s it:
```
openat(AT_FDCWD, ".../user_scripts/wasm/m1.wasm", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 72
write(72<.../m1.wasm>, "\0asm\1\0\0\0...", 120) = 120
# no fsync/fdatasync on fd 72, and no fsync of the containing directory
```
`CREATE FUNCTION` persists its `.sql` durably:
```
openat(AT_FDCWD, ".../user_defined/function_isp.sql.tmp", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 62
fdatasync(62<.../function_isp.sql.tmp>) = 0
renameat2(AT_FDCWD, ".../function_isp.sql.tmp", AT_FDCWD, ".../function_isp.sql", RENAME_NOREPLACE) = 0
```
Where this happens in the code
- `WasmModuleManager::saveModule` — `src/Interpreters/WasmModuleManager.cpp:194-197`: `writeFile(..., WriteMode::Rewrite)` (opens `O_TRUNC`), `write`, `finalize()`, with no `sync()`, no write-to-temp-then-rename, and no directory `fsync`.
- Startup registration — `registerExistingModules` / `validateModuleFile` (`WasmModuleManager.cpp:274-300`, `:107-127`): a file shorter than 16 bytes is silently ignored.
- Re-upload of a torn module throws `FILE_ALREADY_EXISTS` (`saveModule`, `:181`).
- `deleteModuleIfExists` — `:256-272`: a bare `removeFileIfExists`, no directory `fsync`.
- The startup crash originates in `UserDefinedSQLFunctionFactory::loadFunctions` → `UserDefinedWebAssemblyFunctionFactory::prepareFunction` → `WasmModuleManager::getModule`, and the exception propagates to `Poco::Util::Application::run`.
### Expected behavior
An acknowledged `INSERT INTO system.webassembly_modules` should be durable: the module file and its directory entry should be `fsync`ed before the statement returns, and the file should be written atomically (write to a temporary name, `fsync`, then rename) so a torn write cannot leave it truncated. A missing or truncated module referenced by a function should be logged and skipped at startup, not abort the whole server.
### Error message and/or stacktrace
```
Application: Code: 117. DB::Exception: Cannot load WebAssembly module 'm1':
File 'wasm/m1.wasm' is too small to be a valid WebAssembly module:
while loading user defined function `isp`. (INCORRECT_DATA)
...
DB::WasmModuleManager::getModule(...)
DB::UserDefinedWebAssemblyFunctionFactory::prepareFunction(...)
DB::UserDefinedWebAssemblyFunctionFactory::addOrReplace(...)
DB::UserDefinedSQLFunctionFactory::loadFunctions(...)
...
Poco::Util::Application::run()
```
### Additional context
This is the same class as the non-atomic `format_version.txt` write (#111553) and the "durable reference to a non-durable referent" ordering issue (#111339), on a new on-disk file family (`user_scripts/*.wasm`), but here the consequence is a full instance outage rather than only data loss. Found with the ClickFawkes crash-durability framework.
Contributor guide
Assessment
This issue has not been assessed yet.