electric-sql / electric-sql/pglite
NodeFS ordered persistence and synchronization extension contract
- Dominant language
- TypeScript
- Stars
- 16k
- Forks
- 442
- Avg merge
- 20h 19m
- Merged PRs (30d)
- 7
Description
We are evaluating PGlite 0.5.8 for a Node application and trying to establish the
supported route for acknowledging persistent writes across an OS crash or power
loss. We found gaps in the released NodeFS path and would appreciate guidance
before building a custom VFS. We have not demonstrated naturally occurring
database corruption or a real power-loss failure.
Version: `@electric-sql/pglite@0.5.8`, upstream
[`ae182ff8bd5ba4acb887d6c925d607a1498aa0b5`](https://github.com/electric-sql/pglite/tree/ae182ff8bd5ba4acb887d6c925d607a1498aa0b5).
Source inspected from the published package and its embedded TypeScript maps:
- Startup defaults include `-F`. NodeFS inherits an empty `syncToFs()`.
- Emitted `_fd_sync` returns success if a stream has no fsync callback; NODEFS
has no such callback. Emitted fdatasync validates the stream and returns
success without a host synchronization call.
- NODEFS acquires native descriptors only for regular-file streams. Its
replacement path attempts a virtual unlink before the host rename.
- The exported BaseFilesystem helper also lacks fsync/fdatasync operations.
Its generated open handler does not pass stream flags to `baseFS.open`, its
unlink handler suppresses errors, and its syncfs implementation is empty.
Public source references: [startup](https://github.com/electric-sql/pglite/blob/ae182ff8bd5ba4acb887d6c925d607a1498aa0b5/packages/pglite/src/pglite.ts),
[NodeFS](https://github.com/electric-sql/pglite/blob/ae182ff8bd5ba4acb887d6c925d607a1498aa0b5/packages/pglite/src/fs/nodefs.ts),
[BaseFilesystem](https://github.com/electric-sql/pglite/blob/ae182ff8bd5ba4acb887d6c925d607a1498aa0b5/packages/pglite/src/fs/base.ts).
The inspected emitted `dist/pglite.js` has SHA256
`d7db324430326d0a2189a4aa10a11214fd0c92fdbf0df5781bb57f947b491707`.
Our existing synthetic traces used Node 24.20.0 on Windows/x64, Linux/x64 and
macOS/arm64. They traced actual engine and host operations through bootstrap,
migration, mutation, receipt reads, checkpoint and close. Positive host controls
verified that synchronization hooks were observable. No complete persistence
path was established. These were capability/fault probes, not a cache-loss model.
| Control | Observed result |
| --- | --- |
| Default NodeFS; explicit fsync settings | No complete host synchronization path established. An awaited hook alone does not establish a barrier. |
| `wal_sync_method=open_datasync` | Linux/Mac WAL opens carried Node's O_SYNC bit; Windows observed O_RDWR and exposed no Node O_SYNC/O_DSYNC constant. This only characterizes that tested path. |
| `wal_sync_method=open_sync` | Startup rejected in stock and traced controls on all three tuples. |
| Direct host directory synchronization | Succeeded on Linux/Mac; EPERM on Windows. This is not a claim that all Windows persistence mechanisms are unavailable. |
| Checkpoint replacement and injected rename EIO | Replacement was observed; crash atomicity was not established. The injected error also exposed a separately diagnosed protocol-health problem described below. |
After a deliberately injected checkpoint rename EIO, our raw vendor controls
observed subsequent receipt SELECTs without Bind/command completion being
represented as empty query results. Fresh-owner reopening after observed prior
termination could read the earlier receipt. We now reject incomplete protocol
observations and stop ordinary use of an unhealthy instance in our application.
This is failure containment, not a repair of the persistent storage path or
evidence that a rejected operation rolled back. We can separate this topic from
the VFS question if that is easier to track.
We reviewed [#1066](https://github.com/electric-sql/pglite/pull/1066),
[#1063](https://github.com/electric-sql/pglite/pull/1063) and
[#1089](https://github.com/electric-sql/pglite/pull/1089). They appear to improve
synchronization scheduling/failure handling and close ordering without supplying
the missing NodeFS file/namespace barriers.
Questions:
1. Is there a supported Node filesystem implementation or planned upstream route
that supplies complete ordered persistence, including bootstrap, WAL/data and
namespace operations, on these platforms?
2. Which extension seam should own flags, descriptor management, both fsync and
fdatasync, error propagation and replacement semantics? Which compiled routes
require upstream changes rather than a BaseFilesystem subclass?
3. What persistence and maintenance contract does upstream intend for the Node
backend, especially Windows namespace operations and close with pending I/O?
The source references and trace summary above are the available report. We have
not attached our application's fixtures or a standalone reproduction. If useful,
we can prepare a reduced vendor-only synthetic example for a specific path and
verify it before sharing it.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading packages/pglite/src/pglite.ts, packages/pglite/src/fs/nodefs.ts, and packages/pglite/src/fs/base.ts at the referenced commit, then compare their behavior with the inspected dist/pglite.js. Reproduce one specific synchronization or checkpoint-replacement path with a reduced vendor-only example; done requires an agreed supported persistence contract or a narrowly scoped upstream change with platform-specific verification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nodejs, postgres, typescript, wasm
- Domain
- backend, database, operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100