HarperFast / HarperFast/harper-pro

repair_blob_data always reports noConnection on multi-worker instances — it runs on the main thread, connections live on http workers

Open
#684 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

> **Suggested priority: P1** — this operation is the only remediation for the diverged records
> that #683 and harper#2134 are actively creating (measured: 25% of sampled serve-path blob
> faults are still repairable from a peer), and it currently cannot repair anything on any
> multi-worker instance. Fixing it hands operators the escape hatch that #683 otherwise denies.

**Area:** replication · **Line refs:** `v5.2.1` · **Seen on:** 5.2.1 (4-node production cluster, multiple http workers)

## Symptom

`repair_blob_data` (the #388 blob backfill, `replication/blobRepair.ts`) returns immediately and
repairs nothing on a normal multi-worker instance:

```
[main/0] [warn] [blob-repair]: Blob repair complete for page_cache { checked: 1, repaired: 0, failed: 0, noConnection: 1 }
```

`checked: 1` shows the sweep DID find an incomplete blob ref; it then called
`getRepairConnectionsForDB('page_cache')`, got an empty list, and bailed
(`No peer connections available for blob repair, stopping`).

## Cause

The operation executes on the **main thread** (note `[main/0]`), but replication subscription
connections live on **http worker threads**: `startOnMainThread` delegates each database's
subscriptions to a worker round-robin (`subscriptionManager.ts` — `nextWorkerIndex`,
`dbReplicationWorkers`), and `replicator.ts`'s module-level `connections` map is per-thread state.
On the main thread that map never contains user-database connections, so
`getRepairConnectionsForDB()` is empty by construction and `repairBlobs()` can never do work when
invoked through the operations API.

In other words: the only supported way to invoke the repair guarantees it cannot repair. It
presumably worked in single-threaded/dev contexts (`threads: 0`, where subscriptions run on this
thread — the `single-threaded instances have no worker assigned` branch), which is how it shipped
looking functional.

`cluster_status` solved this same main-vs-worker split by reading per-(db,peer)
SharedArrayBuffers; `blobRepair` has no equivalent bridge.

## Suggested fix

Route the operation to (one of) the worker thread(s) owning the target database's subscription
connections — e.g. post a message to the worker recorded in `dbReplicationWorkers` and have it run
`repairBlobs` there, returning progress via the existing log channel. Alternatively let
`repairBlobs` build dedicated repair connections instead of borrowing subscription connections —
which would also remove the documented side effect that a blob-save failure during repair sets
`hasBlobGap` on the live subscription connection (`getRepairConnectionsForDB` doc comment, #368).

## Context

Found while working harper-pro#683 (hasBlobGap one-way latch): the repair sweep is the natural
operator remedy for the placeholder files that drive #683's latch, and it currently cannot be
executed against a production topology.

Contributor guide

Open the contributing guide

Research direction

Read replication/blobRepair.ts, then trace getRepairConnectionsForDB and repairBlobs through replicator.ts. Compare that path with subscriptionManager.ts, especially nextWorkerIndex and dbReplicationWorkers, and review how cluster_status bridges main and worker state. Done means repair_blob_data can repair an incomplete blob reference on a multi-worker instance instead of reporting noConnection.

Written by the indexing model from the issue text.

Assessment

Tech stack
nodejs, typescript
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.