electric-sql / electric-sql/pglite
WASI build: setjmp/longjmp error recovery works under standardized wasm EH (wasi-sdk 33 + wazero) — findings and patches from embedding pglite.wasi in Go
- Dominant language
- TypeScript
- Stars
- 16k
- Forks
- 442
- Avg merge
- 20h 19m
- Merged PRs (30d)
- 7
Description
## Summary
We embedded the `REL_17_5_WASM-pglite` WASI build (branch head `2194acf`, built with the in-tree `wasm-build.sh` + portable-sdk 3.1.74.12.0) in a Go host under wazero, and got **full PostgreSQL ERROR recovery working in the WASI flavor**: erroring queries come back as normal wire-level errors with correct SQLSTATEs and the same instance keeps serving. Our stress gate — 120 mixed error/success iterations, aborted-transaction semantics (25P02 → ROLLBACK), a 1000-error storm on one instance, deep-recursion limits — passes 200 consecutive runs with real pgx clients.
Along the way we found and fixed several issues that we believe affect *any* consumer of the WASI build, not just ours. Everything below is maintained as a small patch series against `2194acf` (`build/patches/0002…0006` in https://github.com/moznion/wasipg, built reproducibly by `build/build.sh`); we'd be glad to turn any of it into PRs if there's interest.
## 1. Enabling sjlj (the headline)
The "sjlj exception handler off" state turns out to be three separate layers; with all three flipped, PG's own `sigsetjmp` recovery works out of the box:
1. **Toolchain**: the portable-sdk's `hotfix/patch.h` (force-included after user CFLAGS) stubs `sigsetjmp` to `return 0` and `siglongjmp` to a no-op. Since PG 17's no-sigsetjmp fallback in `c.h` is Windows-only, these stubs are the only definition the build ever sees. We rewrite them to real `setjmp`/`longjmp` when a `WASIPG_SJLJ` macro is set (this has to happen in patch.h itself —a later `-include` cannot win the macro-redefinition race).
2. **`src/backend/utils/error/elog.c`**: the `#if defined(__wasi__)` ERROR → `abort()` in `errfinish` (patch 0002; also makes the non-sjlj abort emit the error report first, so hosts can at least see *why*).
3. **`pglite-wasm/pgl_sjlj.c`**: `#if defined(__wasi__)` compiles out the whole sigsetjmp block *including the `PG_exception_stack` hookup* (the code that prints "sjlj exception handler off"), so every ERROR is promoted to FATAL by `pg_re_throw`'s NULL-stack branch (patch 0005).
Build flags: `-mllvm -wasm-enable-sjlj -mllvm -wasm-use-legacy-eh=false` plus `-lsetjmp` at link (patch 0004), compiled with **wasi-sdk 33** (LLVM 22). The `-wasm-use-legacy-eh=false` part matters: LLVM's default legacy EH opcodes are rejected by runtimes that implement the *standardized* exnref proposal (wazero ≥ 1.12 in our case; the legacy opcodes are on their way out generally). wasi-sdk 25's LLVM 19 cannot emit exnref sjlj at all, hence the toolchain bump.
Measured cost of sjlj on the module: **+0.2% size, no measurable boot or per-query latency change.**
## 2. Silent shadow-stack overflow (worth fixing for every consumer)
The `wasi-c` wrapper hardcodes `-z stack-size=131072` *after* user flags, so builds get a 128 KiB shadow stack that cannot be overridden from build scripts. Combined with the `--no-stack-first` layout, those 128 KiB sit directly **above the data section** — PostgreSQL's parser/planner recursion overflows into global data long before PG's `max_stack_depth` check (2 MB default) can trigger, silently corrupting arbitrary state. We saw intermittent traps at completely unrelated sites (relcache hash, ProcArray, ResourceOwner) until we raised the wrapper's stack size to 8 MiB; after that, deep recursion surfaces as PostgreSQL's clean, recoverable stack-depth/parser-depth ERROR. Recommendation: make the stack size overridable (or default it much higher for the PG build), or use `--stack-first` so overflow traps instead of corrupting.
## 3. `freopen(stdin)` in `pgl_initdb` breaks under wazero and newer wasi-libc
Two independent problems with the initdb boot-script replay (`freopen(IDB_PIPE_BOOT, "r", stdin)`):
- wazero's `fd_renumber` refuses stdio targets (it marks fds 0–2 as preopens), so wasi-libc's freopen (open + dup3) fails — and musl's failure path then closes stdin outright. The replay dies with `input in flex scanner failed`. (Reported separately to wazero: https://github.com/wazero/wazero/issues/2518)
- wasi-libc 33 declares `stdin` as `FILE *const`, so the existing `stdin = fdopen(saved_stdin, "r")` restore no longer compiles.
Patch 0003 redirects at the fd level instead: `close(0); open(...)` lands the script on fd 0 via lowest-free allocation, no stream reassignment needed. Works under both wazero and wasmtime.
## 4. Wire-loop findings in `interactive_one` (WASI socket-file transport)
Driving the module hard with a real pgx client surfaced several issues in the per-tick wire loop (all addressed in patch 0006):
- **The tick prologue is not idempotent**, so hosts must not call `interactive_one` as a "poll" when they merely want pending output: the prologue resets MessageContext (freeing memory still referenced by a half-processed extended-protocol batch), invalidates the catalog snapshot, and consumes the `send_ready_for_query` flag. A host-scheduling-dependent number of poll ticks made identical runs diverge into traps, hangs and off-by-one reply pairing. We added a **side-effect-free `wasipg_flush` export** (emit the deferred ReadyForQuery, rename buffered reply bytes onto the reply file, touch nothing else) and drive real ticks only to consume input; something like it would be a great official addition to the ABI.
- The idle-tick block that real `PostgresMain` uses to *send* ReadyForQuery only cleared the flag — if the previous tick could not emit RFQ (auth in progress), the deferred RFQ was silently dropped.
- `wire_flush` skipped the `PGS_OLOCK → PGS_OUT` rename when the `sockfiles` flag was momentarily false while reply bytes were buffered; the next tick's `fopen(OLOCK, "w")` then truncated the stranded reply.
- Bytes left in the pq receive buffer by a partially-processed batch (error mid-pipeline) were never drained by later ticks — the tail, often the Sync, never executed and the client hung. Empty ticks now route into the wire loop when leftover pq bytes exist.
- The recovery sigsetjmp armed at the `incoming:` label, *after* io_init / snapshot invalidation / startup-auth processing — an ERROR raised in that window threw a longjmp no handler in that invocation had registered: uncaught exception, instance dead. The arming now happens at the top of the tick.
Two host-facing semantics that cost us time and may be worth documenting for binding authors:
- After error recovery, `pq_comm_reset` can leave libpq's send accounting in a state where a later flush **re-emits an already-delivered reply**; a host that trusts the byte stream blindly gets its request/reply pairing shifted by one. We guard host-side (each exchange is sealed at its ReadyForQuery).
- The auth handshake exchange legitimately ends *without* ReadyForQuery (e.g. `AuthenticationMD5Password` awaiting the client), so "wait for Z" is not a valid completion rule during startup.
## Repro / artifacts
Everything is reproducible from a clean checkout of https://github.com/moznion/wasipg: `./build/build.sh` builds the bundle from pinned sources (branch `REL_17_5_WASM-pglite` @ `2194acf`, portable-sdk 3.1.74.12.0, wasi-sdk 33 overlay; every input sha256-pinned), and `go test ./conformance/` runs the error-recovery suite with a real pgx client under wazero. Happy to rebase the patch series onto whatever branch is most useful, split it into PRs, or share more traces.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with build/build.sh and the referenced build/patches/0002…0006, then run go test ./conformance/ with the pinned WASI sources and wazero setup. Review the changes involving elog.c, pglite-wasm/pgl_sjlj.c, pgl_initdb, and the interactive_one wire loop. Done means the relevant fixes are separated into project-appropriate changes, reproducible builds pass, and the conformance error-recovery suite remains green.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, go, postgres, wasm
- Domain
- backend, build-system, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100