koala73 / koala73/worldmonitor
chore(forecast): drain the 144-entry pending-judge backlog before age-out (config-only)
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
Part of #7066. This is a time-sensitive production runbook. #7068 owns the permanent fix.
## Goal
Recover as much of the current pending-judge cohort as possible without converting the backlog into VOID, losing item-level accountability, or restoring the known stranding condition afterward.
## Verified baseline
- 144 pending-judge entries; 91 are past deadline.
- Required coverage starts at `deadline - 7d`; the default maximum archive lookback is 14d.
- The oldest overdue deadline is about 26.8d past, so the oldest required start is about 33.8d past.
- The archive reader also has a 15,000-hash limit. A 35d time setting alone does not prove complete coverage.
- Queue depth can fall because entries resolve, VOID, or are reclassified. `pendingJudge < 20` alone is not success.
## Preconditions
- #7068 attempt-level archive/judge instrumentation is deployed.
- The exact target service and environment are identified.
- The current service variables and deployment are captured for rollback without printing secret values.
- No other operator is changing the service during the run.
## Runbook
### 0. Freeze the cohort
Before any configuration change, save the 144 resolution-window keys (`id@deadline`) with deadline, current status, attempt count, and required archive start. Record aggregate counts only in public comments; keep raw production records in the approved private evidence location.
### 1. Prove archive coverage
Set `FORECAST_RESOLUTION_JUDGE_EVIDENCE_MAX_LOOKBACK_MS=3024000000` (35d) in a reversible service-config change, but do not increase throughput yet.
Run a coverage-only probe using the same archive read path. Stop unless:
- `archiveInput.incomplete` is false;
- coverage starts at or before every frozen entry's required start;
- the 15,000-hash limit did not truncate the requested range;
- read time fits the service timeout with headroom.
If coverage is incomplete, do not drain. Adjust the retrieval strategy/limit under #7068 and repeat the probe.
### 2. Canary
Run one normal-cap canary. Reconcile every attempted frozen key and inspect attempt classes.
Stop and roll back throughput changes if any of these occurs:
- any new `judge_retry_exhausted`;
- archive coverage becomes incomplete;
- more than 10% of attempted entries become terminal VOID;
- provider/time budget leaves the run incomplete;
- the expected attempt records are missing.
### 3. Controlled drain
Only after a passing canary:
- set `FORECAST_RESOLUTION_JUDGE_MAX_PER_RUN=48`;
- raise `FORECAST_RESOLUTION_JUDGE_RUN_BUDGET_MS` only within the verified service timeout;
- adjust archive timeout only from observed probe timing;
- trigger one run at a time and reconcile its exact attempted keys before the next run.
Use the repository's supported Railway JSON-patch path with the exact service ID. Do not use a dot-path edit that can silently no-op.
### 4. Stop and restore safely
Stop when every frozen entry is reconciled or explicitly recorded as still pending with its blocker. Queue depth is secondary.
Restore the temporary throughput and run-budget overrides on success or abort. Keep the safe maximum lookback until #7068's permanent archive-horizon fix and alert are deployed; do not restore the known stranding condition merely because the drain ended.
Verify the live service configuration after restoration.
## Acceptance
- [ ] The exact 144-entry cohort is frozen before mutation.
- [ ] Coverage-only probe proves all required windows, hash-limit completeness, and timeout headroom.
- [ ] One normal-cap canary passes before throughput rises.
- [ ] Every run has item-level before/after reconciliation and attempt-class totals.
- [ ] All frozen entries are accounted for as YES, NO, VOID by reason, or still pending with a precise blocker.
- [ ] No success claim is based only on queue depth.
- [ ] Temporary throughput/budget variables are restored on success or abort.
- [ ] Safe lookback remains until #7068 prevents recurrence; final live configuration is verified.
- [ ] Before/after totals and a redacted outcome split are posted to the issue.
## Rollback
On any stop condition, restore the saved non-secret configuration values, verify the service, stop further manual triggers, and attach the attempted-key reconciliation to #7068. Do not retry the full batch blindly.
Contributor guide
Research direction
Start by reading #7068 and verify the exact target service, environment, saved non-secret configuration, and supported Railway JSON-patch path. Run the coverage-only archive probe before changing throughput, then perform the canary and reconcile each attempted key. Done means the frozen 144-entry cohort is accounted for, temporary overrides are restored, safe lookback remains, and live configuration is verified.
Written by the indexing model from the issue text.
Assessment
- Domain
- devops, infrastructure
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 20/100