HarperFast / HarperFast/harper
sourcedFrom origin-failure handling: no fetch timeout (unbounded hang) + single-flight fans out to N on failure
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
A `sourcedFrom` cache table over an external origin has two failure-handling gaps that bite hardest exactly when the origin is unhealthy:
1. **No Harper-side origin-fetch timeout → unbounded request hang.** The cold-read populate path (`resources/Table.ts` `getFromSource`/`throttledCallToSource`) `await`s the source resource's `get()` with no Harper-side time bound. A non-responding origin hangs the Harper request indefinitely (the only bound is whatever timeout the source adapter sets itself). `LOCK_TIMEOUT` (~10s) bounds only stampede *followers'* lock wait, not the in-flight leader fetch.
2. **Single-flight collapses to N-fan-out on origin FAILURE.** The cold-read stampede dedup (tryLock) collapses waiters to a single origin call only when the leader **succeeds and writes**. On a 500 or a timeout, each waiter re-enters `getFromSource` and re-fetches → 50 concurrent cold reads of a 500ing origin produce 50 origin hits (or 50 hangs). A thundering-herd amplifier precisely when the origin is already sick.
## Severity
Medium — availability/DoS-shaped under an origin outage (a common production event): request handlers can pin indefinitely, and load multiplies on the failing origin. Mitigated only if the source adapter sets its own fetch timeout; Harper provides no backstop. No data loss.
## Repro
Cache table `Cache` `sourcedFrom` an in-test origin you control (a small `http.createServer`), cold reads via the populating REST PK GET (`GET /Cache/`):
| origin behavior | result |
|---|---|
| 500 | HTTP 500, fast/bounded ✅ |
| connection refused | HTTP 500 (ECONNREFUSED), fast ✅; serves stale on refresh ✅ |
| slow (responsive) | bounded by origin latency ✅ |
| **never responds** | request **hangs** (no server-side bound) ❌ |
| 50× concurrent cold read, origin 500 | **50 origin hits** (no dedup on failure) ❌ |
| 50× concurrent cold read, origin hangs | **all 50 hang** ❌ |
Successful cold-read stampede correctly dedups to **1** origin hit.
Related (`D-059`, doc/behavior): stale-if-error serves stale on **connection-refused** but **not** on origin **HTTP 500** (the 500 path requires `context.staleIfError`, which REST doesn't set) — graceful degradation is asymmetric across failure classes.
## Recommendation
1. Add a configurable Harper-side origin-fetch timeout (a backstop independent of the adapter) returning a bounded 504/error.
2. Hold the single-flight lock across failure so waiters share the leader's failure/timeout instead of each re-fetching (collapse the fan-out on failure too).
3. Consider honoring stale-if-error for HTTP error responses, not just connection errors (the D-059 asymmetry).
---
*Surfaced by the QA-explorer campaign against Harper `7aaa5a152`. The cited code paths are unchanged vs `main` @`6797f091d` (the only feature-branch delta is the sourcedFrom *subscribe* revalidation gate, a different locus). Ready-to-promote failure-mode test exists. Filed by Claude (Opus 4.8) for @kris.*
Contributor guide
Research direction
Read `resources/Table.ts`, focusing on `getFromSource` and `throttledCallToSource`, and trace the cold-read populate path and its `tryLock` behavior. Use the issue’s controlled `http.createServer` repro with a never-responding origin and concurrent reads against an origin returning 500. Done means origin fetches are bounded and concurrent failures do not each trigger a new origin request; consider the separate stale-if-error behavior noted in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100