P1: Make analysis_zip.lock handling crash-safe with TTL/ownership metadata

Problem

Workers acquire analysis_zip.lock on the NFS shared volume before downloading analysis input data. If a worker crashes or is killed after acquiring the lock but before writing analysis_zip.receipt, the lock file persists indefinitely. All subsequent workers for that analysis enter an infinite wait loop in run_simulate_data_point.rb (~line 401) and make no progress.

During the 2026-08-04 incident, 44+ stale lock files were found, trapping hundreds of workers across multiple analyses simultaneously.

Current behavior

• Worker A creates analysis_zip.lock
• Worker A crashes before creating analysis_zip.receipt
• Workers B, C, D... see the lock and wait up to 8 hours for a receipt that will never appear

Proposed solution

Replace the current lock/receipt mechanism with a crash-tolerant design. Options (pick one or combine):

1. Lock file includes owner metadata + timestamp — subsequent workers can detect locks older than a safe threshold with no heartbeat and reclaim them
2. Redis-based lease — use SET NX PX with a TTL; the lock expires automatically if the owner dies
3. Atomic rename staging — download to a temp path, rename atomically; no lock file needed
4. Heartbeat file — owner updates a heartbeat timestamp; waiters check staleness

At minimum: if a lock exceeds a bounded age and no receipt appears, a subsequent worker should be able to validate and reclaim it safely rather than waiting indefinitely.

Acceptance criteria

• A worker crash mid-download does not permanently block subsequent workers for the same analysis
• Lock reclaim logic is explicit, tested, and bounded in time
• No shared-volume filesystem surgery required during normal operations

Context

• Incident: 2026-08-04 OpenStudio Server queue stall
• Related: reduce initialize_worker_timeout (see companion issue)

Start by reading the lock and receipt handling in run_simulate_data_point.rb around line 401, then trace how workers acquire, wait for, and observe analysis_zip.lock and analysis_zip.receipt on the shared volume. Choose and document a crash-safe ownership or staging design, add explicit bounded reclaim tests, and verify that a mid-download worker failure no longer blocks subsequent workers indefinitely.