HarperFast / HarperFast/harper

Models backends: no retry/backoff on transient failures, and @embed surfaces an unhelpful sanitized error

Open
#1,594 0 comments 0 reactions 1 assignee Claimed by @heskew View on GitHub
area:components bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

Splitting two resilience/DX gaps out of #1593 so they can be tracked and fixed independently of the model-name resolution bug. Both make `@embed` fragile in practice, especially for bulk embedding.

### Environment

- Harper **5.1.15**, Node v24.16.0
- `openai` backend (also reproduces against the Gemini OpenAI-compatible endpoint)

---

## 1. No retry / backoff on transient failures

`components/openai/index.ts#post` issues a single `fetch` and throws immediately on any non-OK status:

```ts
const res = await this.#fetch(`${this.#baseUrl}${path}`, { method: 'POST', headers, body, signal });
if (!res.ok) {
throw new OpenAIBackendError(`OpenAI ${path} returned HTTP ${res.status}${await readErrorSuffix(res)}`);
}
```

There is no handling for `429` / `5xx`, no honoring of `Retry-After`, and no backoff anywhere in the embed path. The only resilience available is:

- `requestTimeoutMs` — a timeout, not a retry.
- Multi-candidate failover in `resources/models/Models.ts` — but only when several backends are configured under one logical name; it fails over to a *different* backend rather than retrying the same one. With a single configured backend (the common case), one transient error throws.

**Why it hurts:** `@embed` computes embeddings **per record at write time** (`resources/models/embedHook.ts`). A single transient provider error (rate spike, blip) therefore fails the whole write — and during a bulk ingest, the whole batch. To ship a working full-corpus embed I had to add retry-with-exponential-backoff in my *ingest client* to compensate for the backend having none.

**Suggested fix:** retry retriable statuses (`429`, `5xx`) with exponential backoff, honoring `Retry-After`, in the backend (or the embed hook). Ideally configurable (e.g. `maxRetries`, `retryBackoffMs`) on the model entry.

## 2. `@embed` surfaces an unhelpful sanitized error

When embedding fails, `embedHook.ts` logs the raw backend error at `error` level but rethrows a generic message:

```
Failed to compute embedding for attribute "embedding"
```

The actual cause — e.g. `OpenAI /embeddings returned HTTP 404: models/default is not found ...` — is only in the server log. During diagnosis this sent me chasing rate limits for a while before I found the log line; the surfaced 500 gave no hint of status code, provider, or model.

**Suggested fix:** include the backend error's HTTP status (and sanitized provider message) in the error surfaced from the `@embed` write, or at minimum the attribute + configured model name, so the failure is actionable without grepping server logs. (Care to keep request content out of it, as the backend already does for its own message.)

---

### Context

Both found while dogfooding the docs-site replatform (semantic search over Harper docs via `@embed`). Related: #1593 (the logical-vs-wire model-name resolution bug that produced the 404 above).

---

## Consolidated from #1595 (closed as duplicate)

#1595 was split from the same #1593 investigation and covered the retry half; folding its specifics here so this issue is the single tracker for both gaps:

- **Retriable surface:** `429` and `5xx` (including Anthropic-style `529` overloaded), plus transient network-level failures. Honor `Retry-After` (seconds or HTTP-date) when present, with a sane cap.
- **Caps:** bounded attempts (2–3 by default), jittered exponential backoff, and the overall deadline must keep respecting `requestTimeoutMs` and the caller's `AbortSignal` — an abort should cancel a pending backoff sleep immediately and never re-attempt.
- **Layer:** the backend layer (shared helper in `resources/models/backendHelpers.ts`) so all callers benefit — `@embed`, `scope.models`, and the `/v1` gateway. Interaction with multi-candidate failover: retry the same backend within its budget first, then fail over to the next candidate as today.
- **Out of scope:** batching/queueing `@embed` bulk writes (separate design if pursued).

### Status note on part 2

The error-surface half was partially improved by #1593's fix: the error rethrown from an `@embed` write now carries the backend error class name and the upstream HTTP status (e.g. `[OpenAIBackendError] (backend HTTP 404)`). Remaining gap: include the configured model name and the backend's already-sanitized provider message so the failure is actionable without grepping server logs.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.