electric-sql / electric-sql/pglite

PGlite wasm: multi-row `INSERT ... ON CONFLICT` under DELETE+INSERT churn probabilistically never returns — main-thread JIT spin, SIGTERM-proof, kill -9 only

Open
#1,068 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
16k
Forks
442
Avg merge
20h 19m
Merged PRs (30d)
7

Description

**Version:** @electric-sql/pglite 0.4.3 (bundled pgvector 0.8.1) · **Runtime:** Bun 1.3.14 · **Platform:** macOS 26.5.2, arm64 (Apple Silicon)

## Summary

Under a sustained stream of per-page `DELETE` + multi-row `INSERT ... ON CONFLICT`
operations (~30 rows per INSERT, 18 columns including a `vector(1024)` column and
a generated `tsvector` column backed by a GIN index), PGlite **probabilistically
enters an infinite loop inside the INSERT execution itself**. The query never
returns, the JS main thread spins at ~99% CPU in JIT-generated code, SIGTERM is
never delivered (the event loop is wedged), and the process requires `kill -9`.

Reproduced **5 times** on 2026-07-25: 4× inside the gbrain embedding pipeline
(with API-driven embedding), and — decisively — **1× in a pure offline SQL
storm with zero network, zero API, zero application code**: just PGlite and raw
SQL against a copy of a real 47k-row database.

## The decisive pure-SQL repro (E5)

Script shape (full script available; pseudocode):

```js
const db = new PGlite('/path/to/copy-of-real-db', { extensions: { vector } });
// 70,555 rows with embeddings across 40,089 pages.
// For each page (20-way Promise-interleaved; PGlite single connection serializes):
// SELECT page's rows (embedding::text)
// DELETE FROM content_chunks WHERE page_id = $1
// INSERT INTO content_chunks (18 cols incl. $n::vector) VALUES (...N rows...)
// ON CONFLICT (page_id, chunk_index) DO UPDATE ...
```

**Result:** froze after ~19,300 of 40,089 pages (~48%), ~80 seconds into the
storm. CPU 97.7%, zero log growth, process wedged until `kill -9`.

Table/index state at freeze time (HNSW vector index had been **dropped**
beforehand — it is NOT required):

```
content_chunks_pkey, content_chunks_stale_idx, idx_chunks_embedding_image,
idx_chunks_embedding_null, idx_chunks_language, idx_chunks_page,
idx_chunks_page_index, idx_chunks_search_vector (GIN FTS, ~104MB),
idx_chunks_symbol_name, idx_chunks_symbol_qualified,
idx_content_chunks_edges_backfill, idx_content_chunks_symbol_lookup
```

## Exclusion matrix (every hypothesis tested live, same database)

| Hypothesis | Result |
|---|---|
| Zero / NaN vectors breaking HNSW | ❌ 100 zero-vectors inserted fine; vectors at freeze all healthy (len=1024, nan=0, inf=0, nonzero=1024, max≈0.2) |
| Statically corrupted index | ❌ full replay of pending inserts exits cleanly |
| Poisoned page content | ❌ freeze pages (4 different ones across runs) re-run individually in seconds |
| Concurrency race | ❌ also freezes with application concurrency = 1 |
| API returning degraded vectors during rate-limit storms | ❌ vector health check at freeze instant: all healthy |
| HNSW index bloat / dead-tuple cliff | ❌ churn raises latency sub-linearly (153→405ms over 1500 rounds), no cliff; **and** a fresh REINDEX'd 438MB index froze at 11%; **and** with `DROP INDEX idx_chunks_embedding` it still froze |
| pgvector / HNSW involvement at all | ❌ **falsified** — freeze reproduced with the HNSW index absent |

The only common factor across all 5 freezes: **a multi-row INSERT (31–33 rows,
with tsvector generated column + 4KB vector column) into the churned table,
executed by the PGlite wasm build**. Page content, index type/presence/age,
concurrency, and vector health are all irrelevant.

## Stack fingerprint (identical across all 5 captures)

`sample(1)` shows 100% of samples on the main thread in a ~50–90-frame chain of
anonymous JIT frames (addresses inside the JSC "JS JIT generated code" region,
which also hosts compiled wasm), via the Bun→JS trampoline (`bun+0x32642b8`).
All other threads idle. Zero open network connections (`lsof`). The inspector
protocol also dies: WS handshake succeeds but `Debugger.enable`/`Debugger.pause`
get zero responses — consistent with a tight JIT/wasm loop with no safepoint.

## Real-world impact

In gbrain (which embeds via exactly this DELETE+INSERT pattern), this manifests
as `gbrain embed --stale` silently freezing mid-run — observed in production at
16% and 65% progress, one process spinning 92 minutes before manual kill.
SIGTERM-based supervisors cannot recover it.

## Suspected areas (unverified pointers)

1. `idx_chunks_search_vector` GIN FTS index — fastupdate pending-list flush
pathology under wasm during high-churn INSERT.
2. PGlite wasm memory growth / executor defect on large multi-row INSERTs with
wide rows (tsvector + vector(1024)).

## Environment

- @electric-sql/pglite 0.4.3 (pgvector extension 0.8.1 bundled; upstream is 0.8.5)
- Bun 1.3.14, macOS 26.5.2 arm64
- Database: single PGlite data dir, ~532MB RSS at freeze, no autovacuum (PGlite),
never VACUUM/REINDEX except where noted
- Forensics archive: 5× `sample` stack dumps, 5× run logs, repro scripts
(happy to share the full bundle + a sanitized database copy)

Contributor guide

No contributing guide indexed for this repository

Research direction

Obtain the full E5 pure-SQL reproduction and sanitized database mentioned in the report, then run it against PGlite 0.4.3 while recording the freeze conditions. Compare runs with the HNSW index absent and with application concurrency set to one, as described in the exclusion matrix. Done means the infinite loop is reproduced and its cause is isolated sufficiently to define a regression test and a scoped fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
bun, postgres, wasm
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.