koala73 / koala73/worldmonitor
ops(forecast): deep-forecast-worker replacement failed while the previous deployment stayed active
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
## Summary
Railway's latest deep-forecast-worker replacement failed before creating an instance. The previous successful deployment remains active and is processing tasks, so this is not a current outage, but production did not adopt the watched-path change and the control plane can look healthy because the old deployment continues serving.
## Production evidence
Observed 2026-07-30 15:21-16:25 UTC.
Failed replacement:
- deployment: 3b59a174-3be5-49dc-b2a5-74b82934c909
- source commit: babeda8d527ddb35fed077449f5cc7e8dba5f703
- status: FAILED
- deploymentStopped: true
- buildOnly: true
- instances: empty
- diagnosis: null
- build logs: none
- deployment logs: none
- initial queued reason: Waiting for build slot
A later current-main commit, e75dd382d03bb80470f5659380a0ddd71bf1d19f, passed Test, Lint Code, Typecheck, Security Audit, and Convex Deploy, but the worker's latest deployment still points at the failed replacement.
Impact is currently masked:
- the previous deployment remains successful;
- its five-minute cron continues running;
- at 16:09 UTC it processed one queued task successfully, patched two forecasts, wrote a trace, and reported quality composite 0.926.
## Problem
A failed changed-path deployment is not operationally accepted merely because an older cron deployment remains runnable. The missing diagnosis and zero logs also leave no actionable root cause for whether Railway timed out in the build queue, failed before build initialization, or hit another control-plane failure.
## Acceptance criteria
- [ ] A fresh deep-forecast-worker deployment from current main reaches SUCCESS.
- [ ] The active deployment runs the expected current watched-path revision.
- [ ] A representative queued deep-forecast task completes and persists its forecast/trace outputs after rollout.
- [ ] The no-instance/no-log failure mode is diagnosed or reproduced sufficiently to distinguish build-slot timeout from code/build failure.
- [ ] Monitoring alerts when the latest required changed-path deployment fails even if an older deployment remains successful.
- [ ] Deployment acceptance records both active deployment success and latest intended revision reachability.
- [ ] A failed build-queue attempt can be safely retried without duplicate task processing.
Related: #4930, #5097.
Contributor guide
Research direction
Start at the deep-forecast-worker deployment and its watched-path and monitoring entry points, then inspect the Railway records for the failed replacement and active revision. Done means a current-main deployment succeeds, a queued task persists forecast and trace outputs, and monitoring distinguishes latest-revision failure from an older active deployment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100