koala73 / koala73/worldmonitor
fix(seeders): resilience bundle follow-ups from #6556 — unmeasured Static timeout, orphaned lock, unbounded laggard phase, per-section starvation
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
Follow-ups deliberately scoped out of the #6556 fix. None of these is currently causing an outage; each is a gap that fix surfaced or left standing.
**Agent note:** these five items are independently shippable. Take **one** numbered section per PR. Do not bundle them.
## 1. `Resilience-Static`'s section timeout is container-honorable but unmeasured
#6556 lowered it 900_000 -> 420_000. The old value was fiction — it sat above Railway's 10-minute container kill, so a long run died by container SIGKILL with its logs lost rather than by the runner's own timeout. 420s is derived from the seeder's design worst case (11 dataset adapters run concurrently via `Promise.allSettled`, each fetch `withRetry(2, 750)` over a 30s timeout, so the slowest chain is ~92s plus a Redis pipeline publish), **not** from an observed run.
The section is interval-gated to 90 days and was last seeded ~45 days ago, so the next real run is roughly a month out — and `RESILIENCE_STATIC_WINDOW_CRON` is `0 */4 1-3 10 *` (Oct 1-3). Before that window:
- Capture an actual end-to-end duration for a full (non-skipped) run.
- If it does not fit comfortably inside 420s, this member does not belong in a bundle sharing a 570s budget — give it its own Railway service with its own cap.
## 2. `seed-resilience-static.mjs` has no SIGTERM handler, so its 2h lock can be orphaned
`LOCK_TTL_MS` is 2 hours and `main()` releases the lock only in a `finally` that a signal bypasses. On timeout the runner sends SIGTERM, the process dies with no handler, and `resilience:static` stays locked for up to 2h.
This is not a regression from #6556 — a container SIGKILL at 600s orphaned the lock identically — but lowering the section timeout to 420s makes the runner's own SIGTERM the path that gets taken, which is the one we can actually handle. Mirror the `process.once('SIGTERM', ...)` + `releaseLock` pattern that `runSeed` callers already get in `scripts/_seed-utils.mjs`.
## 3. `Resilience-Scores`' laggard warm-up has no aggregate deadline
The individual laggard path warms `stillMissing` countries in batches of 5, each request bounded at 30s. With up to 196 countries that is ~20 minutes worst case — longer than any timeout that fits the container cap, so the phase is only ever bounded by being killed.
Give it its own wall budget the way `seed-food-stocks.mjs` does (`FAOSTAT_STAGE_BUDGET_MS` / `fetchPhaseTimeoutMs`): compute a deadline with headroom below the section timeout, stop warming when it passes, and still publish the ranking + intervals for what did warm. Today a degraded run is SIGTERM'd mid-warm and publishes nothing, which is the worse outcome.
## 4. A section can be starved on every tick while the bundle still reports success
The new `starvedTick` guard fires only when the whole tick published nothing (`ran === 0 && deferred > 0 && gracefulFailed === 0`). It does not cover the partial case: a section that fits the budget on its own but sits behind a sibling that consumes enough of it to squeeze the section out on **every** tick. `ran > 0`, so the bundle exits 0 and the deferral looks like ordinary pressure.
`seed-bundle-resilience` is not currently exposed — `Resilience-Scores` warm-runs in ~5.7s, leaving room for either sibling — but the shape is one bad measurement away, and it is exactly #6556's failure mode scoped to one member. `seed-bundle-static-ref`'s `Military-Bases` (550s worst case of a 570s budget, 6th of 7 sections) is the closest real instance.
The signal already exists: at deferral time the runner has the section's `seed-meta` age. A deferral whose age exceeds some multiple of its own `intervalMs` is a stall, not pressure, and should be loud regardless of what else ran.
## 5. `tests/helpers/bundle-section-parser.mjs` residuals
The gate now fails closed on everything below, so these are correctness-of-coverage gaps, not silent passes. Each currently costs a loud, slightly confusing failure if someone writes the shape:
- `stripLineComments` is a string scanner, not a JS tokenizer: it cannot distinguish a regex literal from a comment, so `const RE = /[//]/` would truncate the file. The gate catches this by comparing `script:` counts across the strip, but the message blames a commented-out section.
- `resolveImportedIdentifier` does not follow `import def, { X } from './y.mjs'` or `export { X } from './y.mjs'` re-exports.
- `extractBundleOption` takes the first textual `:` match with no word boundary; the gate compensates by asserting exactly one `maxBundleMs:` per bundle.
If bundle config ever grows beyond flat literals, replace the scanner with a real parse rather than extending the regexes.
Context: #6556 (the outage), #6531 (introduced the budget), #6483 (previous silent stall of the same service, different cause).
Contributor guide
Research direction
Choose one numbered section for a separate PR, as requested. For the lock work, read seed-resilience-static.mjs and mirror the SIGTERM pattern in scripts/_seed-utils.mjs; for the laggard budget, compare its phase with seed-food-stocks.mjs. The parser work is in tests/helpers/bundle-section-parser.mjs, while starvation involves seed-bundle-resilience and seed-meta timing; run the relevant tests and confirm the described failure mode is covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, typescript
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100