Agent work is lost on abnormal stage termination — watcher pushes but never checkpoints uncommitted work

Summary

When a stage terminates abnormally (OOMKill, timeout, pod eviction, cancel), all work the agent did in that stage is silently lost because it was never committed. The filesystem watcher pushes on every change but only pushes committed history; the agent commits just once at end-of-stage, so anything before that final commit is discarded on a hard kill — and nothing records that the stage did any work.

Observed

In an AgentWorkflowRun e2e (Java EE → Quarkus), the verify stage applied 12 compile-fix edits (ShippingService, ShoppingCart, CartEndpoint, OrderEndpoint, ProductEndpoint, application.properties, …), then was OOMKilled during a blackbox mvn quarkus:dev.

Result: 0 commits from verify on the branch, and no record of the edits anywhere in status. The three prior stages (questionnaire/plan/execute), which each reached their end-of-stage commit, persisted fine.

$ kubectl logs <run>-verify -c agent | grep -c "tool: edit ·"   # 12
$ kubectl logs <run>-verify -c agent | grep -c "git commit"     # 0

Mechanism

• harness/internal/watcher/watcher.go — doPush() only calls pushFn; it never commits.
• harness/internal/git/git.go:216 — Push skips when HEAD == baseSHA (no commits produced; skipping push).
• The agent is the sole committer, once, at stage end (via the ## Working Guidelines prompt). SIGKILL (OOM) can't be trapped, so a deferred "final commit" on exit wouldn't run either.

Net effect: the watcher gives a false sense of safety — it reports watcher push: success on every change while actually persisting nothing, because there is nothing committed yet.

Asks

1. Checkpoint uncommitted work. Have the harness periodically (and/or on the watcher's debounced events) git add -A && git commit a WIP checkpoint before pushing, so persisted state tracks the agent's actual progress and survives a hard kill.
2. Report work-in-progress on failure. Surface what the stage produced (files changed / last checkpoint) in the AgentRun status instead of a bare failure, so a killed stage isn't a black hole.

Design tension

This touches the ADR 0007 / ADR 0010 "agent commits locally, harness pushes" contract — harness-authored checkpoint commits are a deliberate deviation (e.g. harness: checkpoint commits that the agent's real end-of-stage commit later supersedes/squashes). Flagging for discussion; may warrant an ADR.

Related secondary gaps (same incident, lower priority)

The above is the primary issue. The OOM that triggered it also exposed two smaller gaps, noted here for context:

• Agent pods run BestEffort — the Agent/AgentRun CRDs expose no resources field and the controller sets no requests/limits, so pods have no memory guarantees and are first to be OOM-killed under node pressure.
• Failure reason isn't surfaced — internal/controller/agentrun_controller.go:644 reports the generic Sandbox finished with reason: PodFailed and never reads the pod's containerStatuses[].state.terminated.reason/exitCode, so OOMKilled (exit 137) is only discoverable by manually inspecting the pod.