P2: Evaluate replacing NFS file-based lock coordination with Redis leases

Problem

The current analysis_zip.lock / analysis_zip.receipt design uses plain files on an NFS shared volume to coordinate which worker downloads and extracts analysis input data. File-based coordination on NFS is fragile:

• Locks persist across worker crashes with no automatic expiry
• NFS does not provide distributed locking semantics
• Lock cleanup requires manual filesystem access during incidents

During the 2026-08-04 incident, 44+ stale lock files trapped hundreds of workers and required manual NFS surgery to resolve.

Proposed solution

Evaluate moving analysis initialization coordination to Redis using a lock-with-TTL pattern:

SET analysis:<id>:init_lock <worker_id> NX PX <ttl_ms>

• Lock expires automatically if the owner dies (no manual cleanup needed)
• Redis is already a required dependency of OpenStudio Server
• Supports atomic operations (SET NX is atomic)
• Receipt status can be stored as a Redis key instead of a file

Scope

• Evaluate feasibility and edge cases (e.g., Redis connectivity loss during initialization)
• Prototype or design the replacement coordination protocol
• Document migration path from file-based to Redis-based coordination
• Implementation tracked separately if approved

Acceptance criteria

• Evaluation document or design proposal produced
• Pros/cons of Redis vs. improved file-based approach documented
• Decision recorded before implementation begins

Context

• Incident: 2026-08-04 OpenStudio Server queue stall
• Redis is already present in the stack

Start by tracing how analysis_zip.lock and analysis_zip.receipt coordinate initialization on the NFS volume, then review Redis's existing role in OpenStudio Server. Compare the proposed SET NX PX lock and Redis receipt against the failure cases described, including Redis connectivity loss. Done means an evaluation or design document records the trade-offs, migration path, and a decision before implementation.