koala73 / koala73/worldmonitor

ops(forecast): deep-forecast-worker replacement failed while the previous deployment stayed active

Open
#5,886 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

## Summary

Railway's latest deep-forecast-worker replacement failed before creating an instance. The previous successful deployment remains active and is processing tasks, so this is not a current outage, but production did not adopt the watched-path change and the control plane can look healthy because the old deployment continues serving.

## Production evidence

Observed 2026-07-30 15:21-16:25 UTC.

Failed replacement:

- deployment: 3b59a174-3be5-49dc-b2a5-74b82934c909
- source commit: babeda8d527ddb35fed077449f5cc7e8dba5f703
- status: FAILED
- deploymentStopped: true
- buildOnly: true
- instances: empty
- diagnosis: null
- build logs: none
- deployment logs: none
- initial queued reason: Waiting for build slot

A later current-main commit, e75dd382d03bb80470f5659380a0ddd71bf1d19f, passed Test, Lint Code, Typecheck, Security Audit, and Convex Deploy, but the worker's latest deployment still points at the failed replacement.

Impact is currently masked:

- the previous deployment remains successful;
- its five-minute cron continues running;
- at 16:09 UTC it processed one queued task successfully, patched two forecasts, wrote a trace, and reported quality composite 0.926.

## Problem

A failed changed-path deployment is not operationally accepted merely because an older cron deployment remains runnable. The missing diagnosis and zero logs also leave no actionable root cause for whether Railway timed out in the build queue, failed before build initialization, or hit another control-plane failure.

## Acceptance criteria

- [ ] A fresh deep-forecast-worker deployment from current main reaches SUCCESS.
- [ ] The active deployment runs the expected current watched-path revision.
- [ ] A representative queued deep-forecast task completes and persists its forecast/trace outputs after rollout.
- [ ] The no-instance/no-log failure mode is diagnosed or reproduced sufficiently to distinguish build-slot timeout from code/build failure.
- [ ] Monitoring alerts when the latest required changed-path deployment fails even if an older deployment remains successful.
- [ ] Deployment acceptance records both active deployment success and latest intended revision reachability.
- [ ] A failed build-queue attempt can be safely retried without duplicate task processing.

Related: #4930, #5097.

Contributor guide

Open the contributing guide

Research direction

Start at the deep-forecast-worker deployment and its watched-path and monitoring entry points, then inspect the Railway records for the failed replacement and active revision. Done means a current-main deployment succeeds, a queued task persists forecast and trace outputs, and monitoring distinguishes latest-revision failure from an older active deployment.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.