Minion should optionally resume in-flight jobs on restart, capped by a retry count

When salt-minion dies mid-job (OOM, SIGKILL, host reboot, crash, container restart), every job with an open proc file at <cachedir>/proc/<jid> is currently discarded on the next start. The proc file itself carries the full dispatched payload (JID, fun, args, kwargs, target, etc.) that would let the minion re-execute the job — but nothing on the restart path acts on it. Callers whose only signal is saltutil.running see the jobs vanish; masters that were waiting on the return time out; nothing gets retried automatically.

Proposal

An opt-in minion config option:

# Number of times a job whose proc file survived a minion crash may be
# resumed on startup. 0 (default) preserves today's behavior (discard).
resume_jobs_on_restart: 0

Semantics:

• On minion start, iterate <cachedir>/proc/*; for each proc file, read the payload, increment a resume_attempts counter stored in the file (or a sibling .attempts file), and:
  • If resume_attempts <= resume_jobs_on_restart → re-dispatch the job locally with the original JID (returners correlate correctly), then rewrite the proc file with the incremented counter.
  • If resume_attempts > resume_jobs_on_restart → drop the job (existing behavior), log at WARNING with the JID and attempts count so an operator can tell resurrection was declined.
• Proc files that succeed complete-and-cleanup as normal, so the counter only survives across a crash, not across a successful run.

Why the cap matters

Without an upper bound, the very job that OOM-killed the minion (or triggered whatever segfault took it down) restarts the minion into the same fatal path forever. resume_jobs_on_restart: 3 gives a job three shots — enough for a transient host issue (memory pressure that eased, a network blip during a state.apply that touched a returner) but bounded so a poison-pill job self-limits.

Interactions worth thinking through

• Master retry (auth_tries, master_alive_interval, publish resend on syndic) — if the master ALSO redispatches on timeout, we could double-execute. The re-dispatch should probably suppress the outgoing return until it's actually a resume path, and/or the master's dedup on (jid, minion_id) should be verified to still fire.
• Scheduled jobs — a scheduled job's proc file that survived a crash is not the same as an ad-hoc publish; resume semantics need to distinguish so we don't re-fire a cron-style schedule.
• Idempotency — states are usually idempotent; runners/wheel/execution modules with side effects (e.g. cmd.run 'reboot') are not. This is an operator's choice to make; the default 0 preserves today's semantics.
• JID uniqueness — re-dispatching under the original JID means the master's job cache sees a single jid with (potentially) a return that arrives long after the original publish timeout. Verify no downstream code assumes job_result_time - job_publish_time < N.

Related code

• Proc-file write sites: salt/minion.py:2728, salt/minion.py:3239
• salt.utils.master.clean_proc_dir() — analogous cleanup on the master side (illustrates the discard model that the minion currently mirrors)
• Existing saltutil.running already reads the same proc files, so an saltutil.resumable counterpart falls out cheaply.

Start with the proc-file write sites in salt/minion.py:2728 and salt/minion.py:3239, then trace the minion restart cleanup path and compare it with salt.utils.master.clean_proc_dir(). Define how the opt-in counter, original JID, scheduled-job distinction, and retry cap fit those paths, and verify the default still discards jobs while declined resumes log the JID and attempt count.