HarperFast / HarperFast/harper

Component-installed setResidency is never evaluated on operations-API writes — placement policy silently unenforced on the ops surface

Open
#2,391 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## What happens

A component installs a record-based residency function on a table:

```js
tables.RecVariant.setResidency((record) => (record.placed ? [] : [server.hostname]));
```

Two write surfaces then persist **different state for the same logical write**, on the same table,
same instance, same worker pool. Measured on current main `374408e3f`, single node, RocksDB:

| Arm | n | outcome |
|---|---|---|
| REST PUT, `threads.count: 1` | 20 | destroyed 20/20 (row rebuilt as `{"tag":"T"}` — indexed attrs only) |
| REST PUT, `threads.count: 4` (spread verified over threadIds 1,2,3,4) | 40 | destroyed 40/40 |
| REST PATCH, threads:4 | 10 | destroyed 10/10 |
| **ops-API `insert`+`update`, threads:4, SAME table** | 20 | **destroyed 0/20**, `appliedNew=20` — full new payload stored |
| **ops-API `insert` of an already-excluded record** | 10 | **`fullyStored=10`, `partialStub=0`** |
| `setResidencyById` control | 20 | destroyed 0/20 (old value survives, new value dropped) |
| positive control (residency includes local host) | 5+5 | full round-trip |

Verbatim, arm (b): `{"id":"b-0007","tag":"T","payload":"SEED-PAYLOAD-b-0007","placed":false}` →
`{"tag":"T"}`. Verbatim, arm (c) on the same table:
`{"id":"c-0000","tag":"T","payload":"FLIP-PAYLOAD-c-0000","placed":true}` — intact.

Two consequences, and the second is the one this issue is about:

1. harper#2257's stated mechanism is wrong. The destruction is **not** thread-dependent — it is
100% at every thread count, and the write verb (PUT vs PATCH) makes no difference either.
A reader of #2257 would expect intermittency; there is none on the REST surface.
2. Data the placement policy excludes is stored locally in full whenever the write arrives on the
operations API — `insert`, `update`, Studio-driven writes, operator writes. The policy is
enforced on one surface and ignored on the other, with nothing logged on either.

## Repro

```bash
cd # SHA 374408e3f, npm run build first
npm run test:integration -- "integrationTests/qa-scratch/qa916-residency-thread.test.ts"
```

Spec + fixture: `state/promote-candidates/P-646/` on the `em-state` branch of skills-internal
(`qa916-residency-thread.test.ts` + `qa916-residency-thread/{resources.js,schema.graphql,config.yaml}`).
No non-default env. Config: two instances, `threads.count: 1` and `threads.count: 4`,
`logging.auditLog: true`. 14/14 green; the run was repeated 4× with identical numbers.

## Mechanism

The component's module graph loads **only on HTTP worker threads**. A marker at fixture top level
prints on threadIds 1,2,3,4 and never 0, while `server/operationsServer.ts:77` states outright:
"The operations API runs only on the main thread". So on the operations thread `setResidency` was
never called and `TableResource.getResidency` is still the built-in static
(`resources/Table.ts:1343-1372`), every list of which begins `[server.hostname]` — the exclusion
branch at `resources/Table.ts:2844-2872` is therefore unreachable there.

Direct confirmation: instrumenting the residency function itself gives **0** invocations for every
ops-API write, against 240 for the REST arm and 60 for the REST PATCH arm.

Four competing explanations were traced and eliminated:
- ops writes *do* reach the same residency block — `insert`/`update` → `upsertRecords` →
`Table.put`/`Table.patch` → `_writeUpdate` (`dataLayer/harperBridge/ResourceBridge.ts:202,262`);
- `residencyId` is not pre-set on that path — it is populated only from `options?.residencyId`
(`resources/Table.ts:2454`), which the ops path never supplies, so the `if (residencyId == undefined)`
guard does run;
- no ITC forwarding to a worker — forwarding happens only for operations absent from the local map
(`server/serverHelpers/serverUtilities.ts:210`), and both operations are present;
- the built-in `getResidency` never excludes the host — array `context.replicateTo` prepends the
hostname if absent and the numeric path starts from it (`resources/Table.ts:1348`); it can return
`undefined`, which skips `if (residency)` entirely.

The JS-resource plugin imports `resources.js` inside `handleApplication` (`resources/jsResource.ts:77`),
and the loader invokes that hook only for worker resources (`components/componentLoader.ts:669`) —
that is the load asymmetry.

## Existing work checked

Searched HarperFast/harper and HarperFast/harper-pro, issues **and** PRs, open **and** closed, for
residency / setResidency / placement. Only harper#2257 (OPEN) is on point; it names the thread-static
scoping in passing but frames it as making the *destruction* intermittent, which this wave measured
and disconfirmed. harper-pro#436 (sharding: residency-vs-routing split), #332 (residency-scoped
metadata visibility) and #201 (sharding residency-function TypeError) are the adjacent family and
none describes the ops-surface reachability gap. Documentation was not found to specify which
surfaces evaluate residency. If the team prefers, this can be folded into #2257 as a comment rather
than filed separately — it is a distinct defect but the same code neighbourhood.

**Prior independent instance of the same structural gap (found in QA state, not the tracker):**
finding **F-124** (2026-07-08, harper `ba12a4741`) established that `server.registerOperation()`
called at a component's `resources.js` top level is permanently unreachable from the ops API,
because the main thread requires `server/operationsServer` while worker threads separately require
`server/serverHelpers/serverUtilities` — Node worker_threads do not share module state, so each
worker holds its own `OPERATION_FUNCTION_MAP` and a component-registered operation lands in a map
the main-thread dispatcher never reads. That is the same asymmetry as this finding, on the
operation-registration path rather than the write path, which is the strongest available support
for "the reachability gap is not residency-specific". F-124 was also never filed.

## What this does NOT prove

Single node, RocksDB only. JS components only — native/Python component styles untested. The
`@source` resolve path (`resources/Table.ts:6051-6076`) is untouched, since the fixture tables have
no source. It does not establish which *other* component-installed table statics are equally
unreachable from the operations thread — only that residency is, and the reachability gap is
plainly not residency-specific. And it does not establish what the correct behaviour is: making the
ops path evaluate residency would extend the #2257 destruction to a new surface, so the fix is
likely gated on #2257's resolution.

---
*From dispatch QA finding F-302, re-verified against harper origin/main e16d9616, 2026-08-29 (componentLoader.ts:815 isWorker gate + operationsServer.ts:77 main-thread-only ops API -- mechanism unchanged; line numbers in the body are from 374408e3f and have drifted slightly).*

Contributor guide

Open the contributing guide

Research direction

Start by running integrationTests/qa-scratch/qa916-residency-thread.test.ts and reading server/operationsServer.ts, componentLoader.ts, resources/jsResource.ts, resources/Table.ts, and dataLayer/harperBridge/ResourceBridge.ts. Trace component loading on worker versus main threads and compare operations-API writes with REST residency evaluation. Done requires an agreed expected behavior, a regression test, and a resolution that accounts for the related #2257 semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
api, backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.