ClickHouse / ClickHouse/ClickHouse

`system.webassembly_modules`: acknowledged module INSERT is not durable (no fsync, in-place O_TRUNC) — power loss loses the module or bricks server startup (Code 117)

Open
#111,646 0 comments 0 reactions 1 assignee Claimed by @fm4v View on GitHub
bug comp-udf minor
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Describe what's wrong

Uploading a WebAssembly module with `INSERT INTO system.webassembly_modules` returns success **before the module file on disk is durable**. The module is stored as `user_scripts/wasm/.wasm`, and that file is written in place (truncate + write) and is never `fsync`ed — neither the file nor its directory.

By contrast, `CREATE FUNCTION ... LANGUAGE WASM` persists the function it creates **durably** (it writes a temporary `.sql`, `fdatasync`s it, then atomically renames it into place). So once both statements are acknowledged, we have a durable function definition pointing at a module file that is not durable.

If the server loses power (or the machine crashes) after those acknowledged statements, two things can happen:

**1. The uploaded module is silently lost.** After the crash the `.wasm` file is empty or truncated, and startup treats a too-short file as "not a valid module" and ignores it. The row disappears from `system.webassembly_modules` even though the `INSERT` was acknowledged.

**2. The server refuses to start at all.** If a `CREATE FUNCTION ... LANGUAGE WASM` referenced that module, the function definition survives the crash but the module does not. On startup ClickHouse loads user-defined functions, fails to load the WASM module, and the exception aborts server startup:

```
Code: 117. DB::Exception: Cannot load WebAssembly module 'm1':
File 'wasm/m1.wasm' is too small to be a valid WebAssembly module:
while loading user defined function `isp`. (INCORRECT_DATA)
```

The server then stays down until an administrator manually deletes the orphaned function `.sql` or the broken module file.

Two related problems in the same code:

- Because the module is written with truncation (not written to a temp file and renamed), a torn write can leave a partially-written module that keeps its 8-byte magic header. Such a module registers on startup but fails to compile when called, and re-uploading the correct bytes then fails with `FILE_ALREADY_EXISTS` — so it cannot be repaired through SQL.
- `deleteModuleIfExists` removes the module file without an `fsync` of the directory, so an acknowledged module deletion can also be undone by a crash.

### Does it reproduce on the most recent release?

Yes, reproduces on `master` (26.7.1.1380). The feature was introduced in v26.3.1, so every release since is affected. Requires the server setting `allow_experimental_webassembly_udf`.

### How to reproduce

Both statements below return success:

```sql
INSERT INTO system.webassembly_modules (name, code, hash)
SELECT
'm1',
base64Decode('AGFzbQEAAAABBgFgAX8BfwMCAQAHDAEIaXNfcHJpbWUAAApUAVIBAX8gAEECSQRAQQAPCyAAQQJGBEBBAQ8LIABBAnBBAEYEQEEADwtBAyEBA0ACQCABIABPDQAgACABcEEARgRAQQAPCyABQQJqIQEMAQsLQQEL'),
reinterpretAsUInt256(unhex('369f6098ed1ac69a97735c26039f090ab78c31d729e8110f086b2ea13611c57d'));

CREATE FUNCTION isp LANGUAGE WASM ABI ROW_DIRECT
FROM 'm1' :: 'is_prime' ARGUMENTS (num UInt32) RETURNS UInt32;

-- kill power to the machine here
```

After the power loss, restart the server: it fails to start with the `Code: 117` error shown above.

I reproduced this with a faithful power-loss simulation: the data directory on an `ext4`/`xfs` `dm-flakey` device, switched to `drop_writes`, then unmounted and remounted so only `fsync`ed data survives — the exact state a power cut leaves. It reproduces 3/3 on ext4 and on xfs. A control file that the test `fsync`s itself survives the same cut, confirming the loss is ClickHouse's and not the test's.

strace showing the asymmetry (no crash required)

The module `INSERT` writes the file in place and never `fsync`s it:

```
openat(AT_FDCWD, ".../user_scripts/wasm/m1.wasm", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 72
write(72<.../m1.wasm>, "\0asm\1\0\0\0...", 120) = 120
# no fsync/fdatasync on fd 72, and no fsync of the containing directory
```

`CREATE FUNCTION` persists its `.sql` durably:

```
openat(AT_FDCWD, ".../user_defined/function_isp.sql.tmp", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 62
fdatasync(62<.../function_isp.sql.tmp>) = 0
renameat2(AT_FDCWD, ".../function_isp.sql.tmp", AT_FDCWD, ".../function_isp.sql", RENAME_NOREPLACE) = 0
```

Where this happens in the code

- `WasmModuleManager::saveModule` — `src/Interpreters/WasmModuleManager.cpp:194-197`: `writeFile(..., WriteMode::Rewrite)` (opens `O_TRUNC`), `write`, `finalize()`, with no `sync()`, no write-to-temp-then-rename, and no directory `fsync`.
- Startup registration — `registerExistingModules` / `validateModuleFile` (`WasmModuleManager.cpp:274-300`, `:107-127`): a file shorter than 16 bytes is silently ignored.
- Re-upload of a torn module throws `FILE_ALREADY_EXISTS` (`saveModule`, `:181`).
- `deleteModuleIfExists` — `:256-272`: a bare `removeFileIfExists`, no directory `fsync`.
- The startup crash originates in `UserDefinedSQLFunctionFactory::loadFunctions` → `UserDefinedWebAssemblyFunctionFactory::prepareFunction` → `WasmModuleManager::getModule`, and the exception propagates to `Poco::Util::Application::run`.

### Expected behavior

An acknowledged `INSERT INTO system.webassembly_modules` should be durable: the module file and its directory entry should be `fsync`ed before the statement returns, and the file should be written atomically (write to a temporary name, `fsync`, then rename) so a torn write cannot leave it truncated. A missing or truncated module referenced by a function should be logged and skipped at startup, not abort the whole server.

### Error message and/or stacktrace

```
Application: Code: 117. DB::Exception: Cannot load WebAssembly module 'm1':
File 'wasm/m1.wasm' is too small to be a valid WebAssembly module:
while loading user defined function `isp`. (INCORRECT_DATA)
...
DB::WasmModuleManager::getModule(...)
DB::UserDefinedWebAssemblyFunctionFactory::prepareFunction(...)
DB::UserDefinedWebAssemblyFunctionFactory::addOrReplace(...)
DB::UserDefinedSQLFunctionFactory::loadFunctions(...)
...
Poco::Util::Application::run()
```

### Additional context

This is the same class as the non-atomic `format_version.txt` write (#111553) and the "durable reference to a non-durable referent" ordering issue (#111339), on a new on-disk file family (`user_scripts/*.wasm`), but here the consequence is a full instance outage rather than only data loss. Found with the ClickFawkes crash-durability framework.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.