HarperFast / HarperFast/harper-pro
sourcedFrom cache tables: concurrent independent resolution can desync a record's metadata from its blob
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
For a `sourcedFrom` (cache) table, when multiple nodes independently resolve the same key
concurrently (e.g. a thundering-herd right after a cluster-wide restart, before any node's
replicated write has landed on the others), the competing resolutions race each other around
the mesh as independent `put`s. A node can end up serving a record whose **metadata and blob
content came from two different, non-corresponding puts** — e.g. `random` settles on the
later-applied put while the blob attribute stays latched to an earlier-applied put's file, or
vice versa. Once this happens it does not self-heal: no further write touches the key, so
nothing re-triggers convergence.
This is a metadata/blob **pairing** consistency gap, distinct from the existing blob-loss/
timeout/orphan-file issues (#386, #388, #411, #412, #481) and from the connection/setup-layer
issues (#641, #642) — the blob transfer itself completes successfully in every observed case;
the bug is that the winning metadata and the winning blob are not guaranteed to come from the
same underlying put.
## Reproduction
`harper-pro/integrationTests/cluster/fullyConnectedReplication.test.mjs`, suite "Deploy app
and test replication", test "Replicating cached blobs". The suite's `before()` hook deploys a
component and restarts all 4 nodes (`deploy_component({ restart: true })`), then waits a flat
10s before the test's first `GET /Location/2`. `Location` is `sourcedFrom` (see
`integrationTests/cluster/fixture/resources.js`) with no proactive push/subscribe wiring, so
each node resolves it lazily and independently on its own first miss — a non-deterministic
source (`random: Math.random()`, a fresh blob per call), so distinct resolutions are trivially
distinguishable.
Temporary instrumentation in `core/resources/Table.ts` (logging `id`, `random`, and the blob's
`getFileId()` at both the local-source-resolution commit site — `getFromSource`'s "Writing
resolved record from source" — and the replication-apply site — `writeUpdate`'s `put` case)
captured, in a single run, two independently-resolved versions of the same record racing each
other as competing replicated `put`s:
```
DBGSRC getFromSource commit id 2 imageFileId 1 random 0.8274337822932474 (node resolves)
DBGREPL writeUpdate type put id 2 imageFileId 2 random 0.7979574033415756 (a different node's competing resolve arrives)
DBGREPL writeUpdate type put id 2 imageFileId 1 random 0.8274337822932474 (the first one arrives here too)
DBGREPL writeUpdate type put id 2 imageFileId 2 random 0.7979574033415756
```
After the race, the test's assertion `bodyFrom1.random === bodyFrom2.random` between two
nodes can pass (metadata converges) while `deepEqual(imageFrom1, imageFrom2)` on
`/LocationImage/:id` fails permanently (confirmed stable-wrong across 60 poll iterations /
15s of continuous re-fetching — never once matched, never self-healed).
## Ask
Determine where `_writeUpdate`'s conflict resolution (`core/resources/Table.ts`) and the
blob attribute's save/reference-swap path (`core/resources/blob.ts`,
`startPreCommitBlobsForRecord`/`trackPersistedBlobs`) can settle on different "winning"
writes for the same record, and make the two consistent — either by making the blob swap
strictly follow the same version comparison the metadata fields use, or by treating a
record+blob update as a single atomic unit that only lands together.
## Related
Surfaced while triaging a first-occurrence nightly failure on `harper-pro` main (Cluster
Integration Tests, Node.js v22 shard, 2026-08-03 nightly run) — see the "Replicating cached
blobs" test failure there. Not #641/#642 (connection/setup layer) or #386/#388/#411/#412/#481
(blob loss/timeout/orphan-file scenarios); this is specifically about metadata/blob pairing
under concurrent independent `sourcedFrom` resolution.
Contributor guide
Assessment
This issue has not been assessed yet.