NatLabRockies / NatLabRockies/OpenStudio-server
P1: Reduce initialize_worker_timeout from 8 hours and add explicit stale-lock failure path
Nobody has claimed this yet.
- Dominant language
- Ruby
- Stars
- 58
- Forks
- 27
- PR merge metrics
- No merged PRs in 30d
Description
Problem
The initialize_worker_timeout defaults to 28,800 seconds (8 hours). When a worker becomes stuck waiting for analysis_zip.receipt due to a stale lock, it occupies a worker slot silently for up to 8 hours before giving up. This turns a transient coordination failure into a long-lived capacity drain.
During the 2026-08-04 incident, workers were observed stuck in the receipt wait loop for 5+ hours with no progress.
Proposed solution
- Reduce the timeout to an operationally safe but much shorter bound (e.g., 15–30 minutes for initialization; keep a separate longer timeout for actual simulation runtime).
- Add explicit stale-lock detection: before entering the wait loop, check if the existing lock is stale (old mtime, no owner heartbeat). If so, emit a structured error and fail fast rather than waiting.
- Emit a structured log event when timing out, including lock age, analysis ID, and reason — making this failure mode visible in logs/metrics.
Acceptance criteria
- Initialization timeout is ≤ 30 minutes (or configurable at a sensible default)
- Worker emits a structured error and exits cleanly when initialization exceeds the timeout
- Stale lock detection short-circuits the wait loop with an actionable log message
- Simulation runtime timeout is not affected by this change
Context
- Incident: 2026-08-04 OpenStudio Server queue stall
- Related: crash-safe lock handling (see companion issue)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate initialize_worker_timeout and the wait loop for analysis_zip.receipt, then inspect how lock age, owner heartbeats, and simulation runtime timeouts are currently handled. Implement the initialization-only timeout, stale-lock failure path, and structured timeout logging described in the acceptance criteria, then verify that workers exit cleanly and simulation runtime timeout behavior is unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ruby
- Domain
- backend, distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100