Sealed manual job can complete after client disconnect but remain permanently INCOMPLETE

What happens

A sealed manual ob job run is executed through the attached client/SSH context. If that context is interrupted during a long job, the Docker workload can continue and complete on the managed host, while Onebox never writes a terminal journal record. ob audit then reports the operation as INCOMPLETE permanently even though the application effect succeeded.

This is more than a display problem: once the controlling process and heartbeat disappear, Onebox can lose authoritative ownership of a still-running data-changing container. A later operation must not infer that the work stopped or permit unsafe overlap merely because the client disappeared.

Observed

Goal production, sealed destructive job catalog-refresh, 2026-09-09:

• Onebox journal start: 2026-09-09T05:32:57Z
• Application dataset ledger created during the run and activated at 2026-09-09T08:38:59Z
• Activated Open Food Facts dataset: 4,275,407 accepted records and 460,192 rejected records
• API /healthz and /readyz both returned HTTP 200 after activation
• No catalog worker remained active
• ob audit still showed operation 20260909-053225-54af549-job_run-219d9e8c182c as INCOMPLETE

The domain evidence proves that the refresh reached activation. The missing terminal Onebox record means the wrapper exit status itself is unknowable after the fact.

Cause in the current lifecycle

RunJobWithJournalID writes the start record, calls the blocking attached runJobPhase(ctx, ...), and only then calls finish(runErr) (internal/engine/job.go). The finish writer uses the same caller context:

runErr := e.runJobPhase(ctx, writer, nil, remoteDir, remoteCompose, "job", []string{job})
return operationID, result, finish(runErr)

and:

if journalErr := writer.Append(ctx, record); journalErr != nil {
    return errors.Join(runErr, fmt.Errorf("journal job finish: %w", journalErr))
}

The underlying command is an attached docker compose run --rm --no-deps (internal/engine/gate.go). Cancellation therefore removes the only actor responsible for appending the finish record. Using the cancelled context for cleanup also means that even a surviving client stack cannot reliably journal interruption.

Expected

A client or SSH disconnect must produce one of two explicit, durable outcomes:

1. the host-supervised job continues, retaining coordination ownership, and records its eventual success or failure independently of the client; or
2. Onebox reliably stops the workload and records an interrupted/unknown terminal outcome.

It must never leave a live or completed data-changing workload behind an indefinitely INCOMPLETE client-owned operation with no reconciliation path.

Suggested minimal direction

Run approved sealed manual jobs under a host-side supervisor, reusing the systemd runner/status machinery already used by scheduled and opt-in durable jobs. The approval, resolved release/runtime digest, data effect, fence, result protocol, and operation ID remain plan-bound; the CLI submits the approved execution and may attach to logs, but is not its lifetime owner.

A bounded non-cancelled cleanup context for the terminal append is still worthwhile, but it is not a complete fix when the CLI process itself exits. The durable authority must live on the managed host.

Do not infer success merely because the container disappeared. Reconcile the supervisor, container identity, result evidence, and journal state; preserve an honest interrupted/unknown state when zero exit cannot be proven.

Progress

Landed — #180 (2128338). One-off job containers now carry ob.operation and
ob.epoch, so a container on the host can be matched to the journal that started it —
nothing downstream is possible without that. An interrupted run records a terminal
interrupted record on a bounded background context, retried once if the caller's context
dies mid-write. And the application lock is held rather than released when an interrupted
run leaves its container alive: ReleaseLock runs on its own background context, so on
Ctrl-C it previously succeeded while the journal append failed, dropping ownership
immediately over a live data-changing container. That correction is worth noting against
this issue's framing above, which attributes the loss of ownership to heartbeat death and
TTL expiry — true for kill -9 and SIGHUP, but for Ctrl-C the lock was gone at once.

Open — #182. The refusal only: ob deploy and ob job run refuse while another
operation's job container is running. It asks Docker rather than reducing the journal,
because a journal records what a client managed to write and the failure this exists for
is a client that did not write. Matching is on operation and epoch — a sealed plan
carries one operation id for its whole life and is re-runnable, and AcquireLock returns
the lock to a caller presenting the id already in it, so a second run of one plan would
otherwise exempt the container its own earlier run left behind.

Withdrawn — #181. It paired the refusal with reconciliation (writing the terminal
record for runs whose client never did). Five review rounds found seven defects, all of
them in the reconciliation half, and a mutation test showed every call site could be
deleted with the suite green. The refusal survived and is now #182; the reconciliation is
back to unstarted below.

Not started — reconciliation. Closing a run whose client never recorded an outcome, so
it stops being INCOMPLETE forever. Worth building, but it wants designing once rather
than repairing repeatedly, and it needs three decisions #181 kept deferring: which
operations reconcile, what an honest terminal record says when the container is --rm and
its evidence is gone, and how ob audit should group a journal whose invocations
interleave.

Not started — host-side supervision. Neither PR makes execution durable. A client
killed outright still records nothing at the time; #181 only lets the next operation
close it honestly. Acceptance items 3, 4 and 5 below are untouched.

Acceptance

• Killing the local CLI or SSH connection during a sealed manual job cannot orphan coordination ownership.
• A continuing host-side job remains protected from overlapping deploys or job runs.
• …and from every other operation that mutates under the same lock. Twenty-one call
  sites take the application lock; two check. ob backup restore and ob destroy are
  the sharpest omissions — restore's own contract says it must not interleave with a
  deploy or a recovery, and restoring a volume under a live migration container is
  exactly that. ob rollback, ob abort, ob service apply, ob exec,
  ob secrets push, ob schedule run and bootstrap are the rest. Rolling a release back while an orphaned migration container is mid-flight is
  the same hazard as deploying over it. The natural fix is to hoist the check to a point
  every mutating path already passes through — AcquireLock is the obvious candidate,
  with care for bootstrap, which takes the lock on a host where Docker may not yet be
  reachable.
• Eventual zero/non-zero exit is durably recorded without the initiating client.
• The operator can reconnect and inspect or follow the same operation ID.
• Tests cover disconnect before container launch, during work, after domain effects but before result commit, and after result commit but before terminal journaling.
• Scheduled-job containers are visible to the same check. They are created by their
  own compose run and carry no ob.operation/ob.epoch, so nothing can see them and
  they run without checking either.
• Migration/destructive approval and plan-staleness guarantees are unchanged.

Related, but not duplicates

• #160 / #161 added durable executions for opt-in scheduled native jobs restricted to data_effect: none; sealed destructive/migration manual jobs remain on the attached path.
• #166 adds a per-job history surface but does not make execution or terminal journaling durable.
• #165 mentioned detached manual runs only as a possible follow-up in a closed, unrelated secret-rotation issue.

Start with RunJobWithJournalID and runJobPhase in internal/engine/job.go, the Docker command path in internal/engine/gate.go, and the existing systemd runner/status machinery. Trace AcquireLock and the operation journal lifecycle before deciding which operations reconcile and what terminal state is honest when container evidence is unavailable. Done means disconnects cannot orphan ownership, durable outcomes remain inspectable by operation ID, and the listed mutation and disconnect cases have tests.