HarperFast / HarperFast/harper

Jobs: no boot-time reconciliation — a job whose worker dies with the process stays IN_PROGRESS forever

Open
#2,635 0 comments 0 reactions 1 assignee Claimed by @cb1kenobi View on GitHub
area:operations-api bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Pre-existing gap in the jobs subsystem; a prerequisite for #2632 (item 3; design note §7.5 in `docs/proposals/archive-restore.md` on branch `design/archive-restore`).

## Problem

`server/jobs/` has no boot-time sweep. Status transitions happen only in `jobRunner` (`IN_PROGRESS`, and `ERROR` on a throw — `server/jobs/jobRunner.ts:108-131`) and in the job worker's own `finally` (`server/jobs/jobProcess.ts:71-88`). A job whose worker dies with the process — crash, `restart`, or `HARPER_EXIT_ON_RESTART` under an orchestrator — stays `IN_PROGRESS` forever. `create_backup` and `restore_backup` already leave such rows today, and `get_job` reports them as running indefinitely.

The restart route for `restore_backup` (item 6) needs to reconcile its own job row at boot; without a generic sweep that would be one-off machinery for one operation.

## Fix

- Record **process-instance ownership** on the job row when it starts: pid plus a per-boot instance id (the `componentPreparationLock` pattern, so pid reuse cannot claim a stale row).
- At boot, mark `IN_PROGRESS` rows whose owner is gone as `ERROR` (or a new `INTERRUPTED` status), with a message naming the interruption.
- Handle rows that exist today with no owner recorded (treat as interrupted when their start predates this boot).
- **Defer to a live restore intent**: a job named by a pending restore intent (item 6) is owned by the boot hook, not interrupted; the sweep must consult that before touching the row.

## Acceptance

- Unit: sweep with and without ownership metadata; a row named by a live intent is left alone; pid reuse with a different instance id is treated as gone.
- Integration: kill Harper mid-`create_backup`, restart, `get_job` shows the job as interrupted rather than in progress.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.