HarperFast / HarperFast/harper
Models backends: no retry/backoff on transient failures, and @embed surfaces an unhelpful sanitized error
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
Splitting two resilience/DX gaps out of #1593 so they can be tracked and fixed independently of the model-name resolution bug. Both make `@embed` fragile in practice, especially for bulk embedding.
### Environment
- Harper **5.1.15**, Node v24.16.0
- `openai` backend (also reproduces against the Gemini OpenAI-compatible endpoint)
---
## 1. No retry / backoff on transient failures
`components/openai/index.ts#post` issues a single `fetch` and throws immediately on any non-OK status:
```ts
const res = await this.#fetch(`${this.#baseUrl}${path}`, { method: 'POST', headers, body, signal });
if (!res.ok) {
throw new OpenAIBackendError(`OpenAI ${path} returned HTTP ${res.status}${await readErrorSuffix(res)}`);
}
```
There is no handling for `429` / `5xx`, no honoring of `Retry-After`, and no backoff anywhere in the embed path. The only resilience available is:
- `requestTimeoutMs` — a timeout, not a retry.
- Multi-candidate failover in `resources/models/Models.ts` — but only when several backends are configured under one logical name; it fails over to a *different* backend rather than retrying the same one. With a single configured backend (the common case), one transient error throws.
**Why it hurts:** `@embed` computes embeddings **per record at write time** (`resources/models/embedHook.ts`). A single transient provider error (rate spike, blip) therefore fails the whole write — and during a bulk ingest, the whole batch. To ship a working full-corpus embed I had to add retry-with-exponential-backoff in my *ingest client* to compensate for the backend having none.
**Suggested fix:** retry retriable statuses (`429`, `5xx`) with exponential backoff, honoring `Retry-After`, in the backend (or the embed hook). Ideally configurable (e.g. `maxRetries`, `retryBackoffMs`) on the model entry.
## 2. `@embed` surfaces an unhelpful sanitized error
When embedding fails, `embedHook.ts` logs the raw backend error at `error` level but rethrows a generic message:
```
Failed to compute embedding for attribute "embedding"
```
The actual cause — e.g. `OpenAI /embeddings returned HTTP 404: models/default is not found ...` — is only in the server log. During diagnosis this sent me chasing rate limits for a while before I found the log line; the surfaced 500 gave no hint of status code, provider, or model.
**Suggested fix:** include the backend error's HTTP status (and sanitized provider message) in the error surfaced from the `@embed` write, or at minimum the attribute + configured model name, so the failure is actionable without grepping server logs. (Care to keep request content out of it, as the backend already does for its own message.)
---
### Context
Both found while dogfooding the docs-site replatform (semantic search over Harper docs via `@embed`). Related: #1593 (the logical-vs-wire model-name resolution bug that produced the 404 above).
---
## Consolidated from #1595 (closed as duplicate)
#1595 was split from the same #1593 investigation and covered the retry half; folding its specifics here so this issue is the single tracker for both gaps:
- **Retriable surface:** `429` and `5xx` (including Anthropic-style `529` overloaded), plus transient network-level failures. Honor `Retry-After` (seconds or HTTP-date) when present, with a sane cap.
- **Caps:** bounded attempts (2–3 by default), jittered exponential backoff, and the overall deadline must keep respecting `requestTimeoutMs` and the caller's `AbortSignal` — an abort should cancel a pending backoff sleep immediately and never re-attempt.
- **Layer:** the backend layer (shared helper in `resources/models/backendHelpers.ts`) so all callers benefit — `@embed`, `scope.models`, and the `/v1` gateway. Interaction with multi-candidate failover: retry the same backend within its budget first, then fail over to the next candidate as today.
- **Out of scope:** batching/queueing `@embed` bulk writes (separate design if pursued).
### Status note on part 2
The error-surface half was partially improved by #1593's fix: the error rethrown from an `@embed` write now carries the backend error class name and the upstream HTTP status (e.g. `[OpenAIBackendError] (backend HTTP 404)`). Remaining gap: include the configured model name and the backend's already-sanitized provider message so the failure is actionable without grepping server logs.
Contributor guide
Assessment
This issue has not been assessed yet.