NatLabRockies / NatLabRockies/OpenStudio-server

P1: Reduce initialize_worker_timeout from 8 hours and add explicit stale-lock failure path

Open
#868 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Ruby
Stars
58
Forks
27
PR merge metrics
No merged PRs in 30d

Description

Problem

The initialize_worker_timeout defaults to 28,800 seconds (8 hours). When a worker becomes stuck waiting for analysis_zip.receipt due to a stale lock, it occupies a worker slot silently for up to 8 hours before giving up. This turns a transient coordination failure into a long-lived capacity drain.

During the 2026-08-04 incident, workers were observed stuck in the receipt wait loop for 5+ hours with no progress.

Proposed solution

  1. Reduce the timeout to an operationally safe but much shorter bound (e.g., 15–30 minutes for initialization; keep a separate longer timeout for actual simulation runtime).
  2. Add explicit stale-lock detection: before entering the wait loop, check if the existing lock is stale (old mtime, no owner heartbeat). If so, emit a structured error and fail fast rather than waiting.
  3. Emit a structured log event when timing out, including lock age, analysis ID, and reason — making this failure mode visible in logs/metrics.

Acceptance criteria

  • Initialization timeout is ≤ 30 minutes (or configurable at a sensible default)
  • Worker emits a structured error and exits cleanly when initialization exceeds the timeout
  • Stale lock detection short-circuits the wait loop with an actionable log message
  • Simulation runtime timeout is not affected by this change

Context

  • Incident: 2026-08-04 OpenStudio Server queue stall
  • Related: crash-safe lock handling (see companion issue)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate initialize_worker_timeout and the wait loop for analysis_zip.receipt, then inspect how lock age, owner heartbeats, and simulation runtime timeouts are currently handled. Implement the initialization-only timeout, stale-lock failure path, and structured timeout logging described in the acceptance criteria, then verify that workers exit cleanly and simulation runtime timeout behavior is unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
backend, distributed-systems, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.