koala73 / koala73/worldmonitor

ops(railway): decommission the quiesced legacy reconcile control plane after the rollback window closes

Open
#7,257 1 comment 0 reactions 0 assignees View on GitHub
area: infrastructure chore P3
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

## Summary

Priority: **P3**, tracking only. #6686 quiesced the legacy Railway reconcile control plane: no schedule, no auto-recovery, both activation flags pinned `false` in normal operation. The runbook (`docs/railway-seed-consolidation-runbook.md`, "Native autodeploy target and quiesced rollback surface") names Railway native autodeploy as the permanent architecture and the legacy control plane as a bounded rollback surface that is **deleted with the old control plane**. Nothing tracked that deletion — #6378 tracked activation, which is now superseded and closed. This issue carries the remainder.

## Preconditions before any deletion

- [ ] Native autodeploy judged stable per the runbook's own bar: `Railway Native Deploy Health` green over an agreed soak window (it is currently red — see the separate watch-path drift issue), and no rollback via the legacy surface during that window.
- [ ] Explicit decision recorded here that the bounded rollback window is over.

## Deletion checklist

Once the rollback window closes, in this order (credentials last so an abort mid-way leaves a working emergency surface):

- [ ] Delete the four legacy workflows: `railway-deploy-trigger.yml`, `railway-deploy-trigger-watchdog.yml`, `railway-reconcile-manual-recovery.yml`, `deploy-railway-reconcile-control.yml`.
- [ ] Delete `workers/railway-reconcile-control/` (Worker + Durable Object) and remove the Cloudflare deployment + DNS for `railway-reconcile-control.worldmonitor.app`.
- [ ] Delete the client scripts and their tests: `scripts/dispatch-stale-railway-reconcile.mjs`, `scripts/railway-reconcile-control-client.mjs`, `scripts/trigger-railway-deploys.mjs` (verify no non-legacy caller first), and the workflow contract tests that pin them (e.g. `tests/railway-deploy-trigger-workflow.test.mjs`).
- [ ] Delete `scripts/configure-railway-native-autodeploy.mjs` — the runbook's own U6 commitment ("temporary and must be deleted in U6 after the bounded migration").
- [ ] Delete the GitHub environments `ingestion-acceptance-production-verification`, `-breakglass`, `-watchdog`, and `railway-reconcile-control-production`, with their HMAC/token secrets. Strip the reconcile secrets from `ingestion-acceptance-production` (keep the environment itself — the ingestion acceptance path still uses it).
- [ ] Delete repo vars `RAILWAY_RECONCILE_CUTOVER_ACTIVE`, `RAILWAY_RECONCILE_AUTO_RECOVERY_ENABLED`, `RAILWAY_RECONCILE_CONTROL_URL`.
- [ ] Revoke the Railway project tokens `github-reconcile-viewer` and `github-reconcile-deploy-v2` in the Railway dashboard.
- [ ] Revoke `RAILWAY_PRODUCTION_TOKEN` after migrating its one remaining workflow dependent (`umami-storage-monitor.yml` — verified 2026-08-28; `seed-freshness-monitor.yml` no longer references it) to the dedicated `VIEWER`-identity credential the runbook prescribes for the permanent monitor. Explicitly carried over from #6378 as non-blocking hygiene.
- [ ] Purge the legacy rollback sections from `docs/railway-seed-consolidation-runbook.md` and update `docs/solutions/integration-issues/railway-seeder-watch-paths-can-skip-deployments.md` where it still points at the reconciler as the compensating control.

## What NOT to delete

- `scripts/check-railway-deploy-drift.mjs`, `scripts/audit-railway-watch-paths.mjs`, `scripts/railway-services.json`, `scripts/railway-native-autodeploy-fleet.json`, and `railway-deploy-drift.yml` — these are the permanent monitor surface, not the legacy control plane.

Refs #6378, #6686, #6325, #6326.

Contributor guide

Open the contributing guide

Research direction

Start with docs/railway-seed-consolidation-runbook.md and confirm the Railway Native Deploy Health soak and explicit rollback-window decision before touching the deletion checklist. Then review the named workflows, workers/railway-reconcile-control/, scripts, tests, environments, variables, and credentials, preserving the files and monitor surface listed under “What NOT to delete.” Done means the legacy control plane and rollback documentation are removed only after all listed dependencies and credentials are safely retired.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
ci-cd, devops, documentation, infrastructure
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.