NatLabRockies / NatLabRockies/OpenStudio-server
P2: Add structured metrics/logging for analysis initialization lock lifecycle
Nobody has claimed this yet.
- Dominant language
- Ruby
- Stars
- 58
- Forks
- 27
- PR merge metrics
- No merged PRs in 30d
Description
Problem
The analysis_zip.lock / analysis_zip.receipt coordination mechanism is opaque. There is no structured observability for lock acquisition, wait duration, reclaim attempts, or timeout exits. This made the 2026-08-04 stale-lock incident hard to detect early and difficult to quantify during response.
Proposed solution
Emit structured log events (key=value pairs or JSON) for each stage of the initialization lock lifecycle:
| Event | Key fields |
|---|---|
| Lock acquired | analysis_id, lock_path, ts |
| Receipt found | analysis_id, wait_duration_s |
| Waiting for receipt | analysis_id, lock_age_s, elapsed_s (periodic) |
| Stale lock detected | analysis_id, lock_age_s, action |
| Timeout exit | analysis_id, elapsed_s, reason |
These events should be parseable by log aggregators (Vector, Loki, etc.) and ideally expose a counter/gauge for alerting on initialization stalls.
Acceptance criteria
- All lock lifecycle stages emit at least one structured log line
- Log lines include enough context to correlate across workers and analyses
- No sensitive data in log output
- Existing log output is not made significantly more verbose under normal conditions
Context
- Incident: 2026-08-04 OpenStudio Server queue stall
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the initialization code that coordinates analysis_zip.lock and analysis_zip.receipt, then trace acquisition, receipt detection, waiting, stale-lock reclamation, and timeout paths. Review the existing logging and metrics conventions before defining structured events with the listed correlation fields. Done means every lifecycle stage is observable, sensitive data is excluded, and normal log volume is not significantly increased.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ruby
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100