Ops rules: Resque silent job loss — mid-ramp dispatch ban + end-of-run sweep

Why

HANDOFF-SCALE-TO-9000.md §R7: jobs die silently when worker children die mid-run (verified again 2026-08-24). This is a process contract, not a bug to patch today — but it's currently tribal knowledge in a handoff doc.

The rules (from the handoff, verified)

1. Dispatch/re-dispatch ONLY after fleet fully converged: readyReplicas == desired AND all mounts ls clean. Mid-ramp dispatch caused the 2026-08-24 orphan wave.
2. After ANY mass recycle: re-dispatch dps with status: started (their children died). queued jobs survive in redis and resume on their own.
3. End-of-run sweep: any dp not completed + completed normal gets one submit_simulation once the fleet is quiet.

Caveats / gotchas to document

• Beware mongoid cursor weirdness: updating while iterating re-emits updated docs (#each) — harmless duplicates, app skips completed-normal, but document it so nobody "fixes" it into a livelock.
• Mongo access: exec into deploy/db, mongosh -u openstudio -p <pass> --authenticationDatabase admin os_docker, collection data_points.

Deliverable

• Runbook section (or standalone doc) covering the three rules
• Optionally: scripted end-of-run sweep tooling so rule 3 isn't hand-run mongosh

Start with HANDOFF-SCALE-TO-9000.md §R7 and compare its three rules with the issue's listed MongoDB and fleet-operation caveats. Add a runbook section or standalone document covering convergence before dispatch, post-recycle redispatch, and the end-of-run sweep; document the cursor behavior and Mongo access details, with sweep tooling optional.