NatLabRockies / NatLabRockies/OpenStudio-server

P2: Add structured metrics/logging for analysis initialization lock lifecycle

Open
#869 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Ruby
Stars
58
Forks
27
PR merge metrics
No merged PRs in 30d

Description

Problem

The analysis_zip.lock / analysis_zip.receipt coordination mechanism is opaque. There is no structured observability for lock acquisition, wait duration, reclaim attempts, or timeout exits. This made the 2026-08-04 stale-lock incident hard to detect early and difficult to quantify during response.

Proposed solution

Emit structured log events (key=value pairs or JSON) for each stage of the initialization lock lifecycle:

Event Key fields
Lock acquired analysis_id, lock_path, ts
Receipt found analysis_id, wait_duration_s
Waiting for receipt analysis_id, lock_age_s, elapsed_s (periodic)
Stale lock detected analysis_id, lock_age_s, action
Timeout exit analysis_id, elapsed_s, reason

These events should be parseable by log aggregators (Vector, Loki, etc.) and ideally expose a counter/gauge for alerting on initialization stalls.

Acceptance criteria

  • All lock lifecycle stages emit at least one structured log line
  • Log lines include enough context to correlate across workers and analyses
  • No sensitive data in log output
  • Existing log output is not made significantly more verbose under normal conditions

Context

  • Incident: 2026-08-04 OpenStudio Server queue stall

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate the initialization code that coordinates analysis_zip.lock and analysis_zip.receipt, then trace acquisition, receipt detection, waiting, stale-lock reclamation, and timeout paths. Review the existing logging and metrics conventions before defining structured events with the listed correlation fields. Done means every lifecycle stage is observable, sensitive data is excluded, and normal log volume is not significantly increased.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
backend, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.