HarperFast / HarperFast/harper

sourcedFrom origin-failure handling: no fetch timeout (unbounded hang) + single-flight fans out to N on failure

Open
#1,398 0 comments 0 reactions 0 assignees View on GitHub
area:rest-api bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

A `sourcedFrom` cache table over an external origin has two failure-handling gaps that bite hardest exactly when the origin is unhealthy:

1. **No Harper-side origin-fetch timeout → unbounded request hang.** The cold-read populate path (`resources/Table.ts` `getFromSource`/`throttledCallToSource`) `await`s the source resource's `get()` with no Harper-side time bound. A non-responding origin hangs the Harper request indefinitely (the only bound is whatever timeout the source adapter sets itself). `LOCK_TIMEOUT` (~10s) bounds only stampede *followers'* lock wait, not the in-flight leader fetch.

2. **Single-flight collapses to N-fan-out on origin FAILURE.** The cold-read stampede dedup (tryLock) collapses waiters to a single origin call only when the leader **succeeds and writes**. On a 500 or a timeout, each waiter re-enters `getFromSource` and re-fetches → 50 concurrent cold reads of a 500ing origin produce 50 origin hits (or 50 hangs). A thundering-herd amplifier precisely when the origin is already sick.

## Severity

Medium — availability/DoS-shaped under an origin outage (a common production event): request handlers can pin indefinitely, and load multiplies on the failing origin. Mitigated only if the source adapter sets its own fetch timeout; Harper provides no backstop. No data loss.

## Repro

Cache table `Cache` `sourcedFrom` an in-test origin you control (a small `http.createServer`), cold reads via the populating REST PK GET (`GET /Cache/`):

| origin behavior | result |
|---|---|
| 500 | HTTP 500, fast/bounded ✅ |
| connection refused | HTTP 500 (ECONNREFUSED), fast ✅; serves stale on refresh ✅ |
| slow (responsive) | bounded by origin latency ✅ |
| **never responds** | request **hangs** (no server-side bound) ❌ |
| 50× concurrent cold read, origin 500 | **50 origin hits** (no dedup on failure) ❌ |
| 50× concurrent cold read, origin hangs | **all 50 hang** ❌ |

Successful cold-read stampede correctly dedups to **1** origin hit.

Related (`D-059`, doc/behavior): stale-if-error serves stale on **connection-refused** but **not** on origin **HTTP 500** (the 500 path requires `context.staleIfError`, which REST doesn't set) — graceful degradation is asymmetric across failure classes.

## Recommendation

1. Add a configurable Harper-side origin-fetch timeout (a backstop independent of the adapter) returning a bounded 504/error.
2. Hold the single-flight lock across failure so waiters share the leader's failure/timeout instead of each re-fetching (collapse the fan-out on failure too).
3. Consider honoring stale-if-error for HTTP error responses, not just connection errors (the D-059 asymmetry).

---
*Surfaced by the QA-explorer campaign against Harper `7aaa5a152`. The cited code paths are unchanged vs `main` @`6797f091d` (the only feature-branch delta is the sourcedFrom *subscribe* revalidation gate, a different locus). Ready-to-promote failure-mode test exists. Filed by Claude (Opus 4.8) for @kris.*

Contributor guide

Open the contributing guide

Research direction

Read `resources/Table.ts`, focusing on `getFromSource` and `throttledCallToSource`, and trace the cold-read populate path and its `tryLock` behavior. Use the issue’s controlled `http.createServer` repro with a never-responding origin and concurrent reads against an origin returning 500. Done means origin fetches are bounded and concurrent failures do not each trigger a new origin request; consider the separate stale-if-error behavior noted in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.