HarperFast / HarperFast/harper

Component load race fails per-worker (200/500 split-brain at threads>1) and is invisible to get_status

Open
#1,951 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

When two components have an undeclared load-time dependency on each other (e.g. component B extends a table class exported by component A, per #1931), the resulting failure is **per-worker**, not per-process: at `threads.count > 1`, each worker thread runs the same undeclared race independently, so the *same URL* can serve `200` from some worker threads and a structured `500` (`ResourceLoadError`) from others — persistently, not just during a boot-time window.

This is a distinct, separate hazard from the declaration-gap #1931 describes:

1. **#1931** is about there being no way to *declare* a cross-component dependency, so boot order is undefined.
2. **This issue** is about the operational blast radius once that race goes badly: the failure is loud per-request (a real `ResourceLoadError` 500, not a silent 404 or phantom 200 — the mitigating half), but it is **invisible to `get_status`**, so an operator/health-check has no signal that some fraction of workers are serving a broken component while others are healthy.

## Reproduction

Fixture: two components, "provider" (defines a table + a custom `Resource` subclass) and "consumer" (extends the provider's table class at JS module top level, with no declared dependency). Booted with `threads.count: 8`.

- Fresh, sequential (`Connection: close`) requests to the consumer's own route (`/ConsumerPing`) show **both `200` and `500`** across the run — i.e. some worker threads won the race, some lost it, and it does not converge after a multi-second settle window.
- Losing workers log a real `ResourceLoadError: ... Class extends value undefined is not a constructor or null` and serve a structured 500 via the component's `ErrorResource`.
- Calling `operation: get_status` (no `id`) during this same split-brain window does **not** reveal the divergence — see the "cross-thread aggregation" gap below, which independently makes even the signal that *does* exist unreliable.

## Root cause found and fixed in a companion PR

Investigating *why* `get_status` never showed the divergence turned up a separate, well-scoped bug in the cross-thread component-status aggregator: `CrossThreadStatusCollector.collect()` (`components/status/crossThread.ts`) computed how many worker responses to wait for from `getWorkerCount()` — which only reports the **calling thread's own same-type pool size** (and defaults to `1` when called from the main thread, since the main thread isn't part of any worker pool). In a normal deployment, `get_status` is served from the main thread while HTTP traffic runs on N worker threads, so the collector was capping `expectedResponses` at `1` and declaring the collection complete after the *first* worker replied — silently discarding every other (and possibly disagreeing) worker's status. In live testing with `threads.count: 8`, this meant `get_status`'s aggregated component status only ever reflected **one out of eight** worker threads, regardless of how many actually disagreed.

That specific bug is fixed in a companion PR (using the already-tracked total worker-thread count from `manageThreads.js`'s `workers` registry instead of the calling thread's own pool size), so `get_status` now correctly aggregates from every live worker thread rather than just one.

## What is *not* fixed here

The underlying cross-component load-order race itself (why the failure happens at all) is unaddressed — that requires the dependency-declaration mechanism #1931 asks for, or some other explicit ordering/retry contract. This issue is scoped to the operational visibility gap (get_status) and the per-worker persistence of the failure; #1931 remains the tracking issue for the actual ordering fix.

Cross-linked to #1931 (same underlying trigger scenario, different aspect of the defect).

Contributor guide

Open the contributing guide

Research direction

Start with components/status/crossThread.ts and the worker registry in manageThreads.js. The issue says the get_status aggregation bug is fixed in a companion PR, while the underlying cross-component load-order race belongs to #1931; verify whether any operational-visibility work remains before making changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.