koala73 / koala73/worldmonitor

chore(forecast): drain the 144-entry pending-judge backlog before age-out (config-only)

Open
#7,069 0 comments 0 reactions 0 assignees View on GitHub
area:seeders chore P1
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

Part of #7066. This is a time-sensitive production runbook. #7068 owns the permanent fix.

## Goal

Recover as much of the current pending-judge cohort as possible without converting the backlog into VOID, losing item-level accountability, or restoring the known stranding condition afterward.

## Verified baseline

- 144 pending-judge entries; 91 are past deadline.
- Required coverage starts at `deadline - 7d`; the default maximum archive lookback is 14d.
- The oldest overdue deadline is about 26.8d past, so the oldest required start is about 33.8d past.
- The archive reader also has a 15,000-hash limit. A 35d time setting alone does not prove complete coverage.
- Queue depth can fall because entries resolve, VOID, or are reclassified. `pendingJudge < 20` alone is not success.

## Preconditions

- #7068 attempt-level archive/judge instrumentation is deployed.
- The exact target service and environment are identified.
- The current service variables and deployment are captured for rollback without printing secret values.
- No other operator is changing the service during the run.

## Runbook

### 0. Freeze the cohort

Before any configuration change, save the 144 resolution-window keys (`id@deadline`) with deadline, current status, attempt count, and required archive start. Record aggregate counts only in public comments; keep raw production records in the approved private evidence location.

### 1. Prove archive coverage

Set `FORECAST_RESOLUTION_JUDGE_EVIDENCE_MAX_LOOKBACK_MS=3024000000` (35d) in a reversible service-config change, but do not increase throughput yet.

Run a coverage-only probe using the same archive read path. Stop unless:

- `archiveInput.incomplete` is false;
- coverage starts at or before every frozen entry's required start;
- the 15,000-hash limit did not truncate the requested range;
- read time fits the service timeout with headroom.

If coverage is incomplete, do not drain. Adjust the retrieval strategy/limit under #7068 and repeat the probe.

### 2. Canary

Run one normal-cap canary. Reconcile every attempted frozen key and inspect attempt classes.

Stop and roll back throughput changes if any of these occurs:

- any new `judge_retry_exhausted`;
- archive coverage becomes incomplete;
- more than 10% of attempted entries become terminal VOID;
- provider/time budget leaves the run incomplete;
- the expected attempt records are missing.

### 3. Controlled drain

Only after a passing canary:

- set `FORECAST_RESOLUTION_JUDGE_MAX_PER_RUN=48`;
- raise `FORECAST_RESOLUTION_JUDGE_RUN_BUDGET_MS` only within the verified service timeout;
- adjust archive timeout only from observed probe timing;
- trigger one run at a time and reconcile its exact attempted keys before the next run.

Use the repository's supported Railway JSON-patch path with the exact service ID. Do not use a dot-path edit that can silently no-op.

### 4. Stop and restore safely

Stop when every frozen entry is reconciled or explicitly recorded as still pending with its blocker. Queue depth is secondary.

Restore the temporary throughput and run-budget overrides on success or abort. Keep the safe maximum lookback until #7068's permanent archive-horizon fix and alert are deployed; do not restore the known stranding condition merely because the drain ended.

Verify the live service configuration after restoration.

## Acceptance

- [ ] The exact 144-entry cohort is frozen before mutation.
- [ ] Coverage-only probe proves all required windows, hash-limit completeness, and timeout headroom.
- [ ] One normal-cap canary passes before throughput rises.
- [ ] Every run has item-level before/after reconciliation and attempt-class totals.
- [ ] All frozen entries are accounted for as YES, NO, VOID by reason, or still pending with a precise blocker.
- [ ] No success claim is based only on queue depth.
- [ ] Temporary throughput/budget variables are restored on success or abort.
- [ ] Safe lookback remains until #7068 prevents recurrence; final live configuration is verified.
- [ ] Before/after totals and a redacted outcome split are posted to the issue.

## Rollback

On any stop condition, restore the saved non-secret configuration values, verify the service, stop further manual triggers, and attach the attempted-key reconciliation to #7068. Do not retry the full batch blindly.

Contributor guide

Open the contributing guide

Research direction

Start by reading #7068 and verify the exact target service, environment, saved non-secret configuration, and supported Railway JSON-patch path. Run the coverage-only archive probe before changing throughput, then perform the canary and reconcile each attempted key. Done means the frozen 144-entry cohort is accounted for, temporary overrides are restored, safe lookback remains, and live configuration is verified.

Written by the indexing model from the issue text.

Assessment

Domain
devops, infrastructure
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.