koala73 / koala73/worldmonitor
epic(news): make digest retention bounded, category-complete, and honest
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
## Goal
Make the news-digest system bounded, category-complete under its declared deadline, and honest when it serves incomplete or stale content.
This epic converts the validated parts of the 2026-08-22 news-triage audit into implementation work. It does not treat every internal artifact as something that must become public.
## Frozen evidence
- Repository reference: `origin/main` at `8aa7b06f7`.
- Production observation: 2026-08-22 around 08:36 UTC, read-only Redis queries.
- The 109 digest-accumulator keys contained 5,289,669 members. 5,185,238 were older than the documented 48-hour digest window.
- `digest:accumulator:v1:full:en` contained 140,197 members. 137,597 were older than 48 hours, and the oldest member was about 135.8 days old.
- The sampled `full/en` digest served 143 items. It reported 113 unfinished feeds, seven empty feeds, and no `intel` category.
- Its coverage ledger reported 4,920 per-feed-cap drops and 352 per-category-cap drops.
- The current server handler has a 10-second overall fetch deadline inside the 25-second initial-response limit. Feeds that never start and feeds that fail after starting are both currently reported as `timeout`.
- The digest response already has `feedStatuses`. The missing contract is aggregate attempt coverage, category state, and stale-content state.
- The forecast resolver currently reads the same accumulator with a seven-day normal and 14-day maximum evidence window. A direct 48-hour cleanup would destroy required forecast evidence.
These values are diagnostic anchors. Do not put mutable Redis values into pull-request pass/fail checks.
## Product contract
1. Digest storage must have an enforced member-level retention bound. Key TTL refresh is not a retention policy.
2. Forecast evidence must remain available for its declared maximum lookback without depending on digest-only retention or a shorter story-track TTL.
3. Every digest build attempt must report whether a feed completed, failed after starting, was aborted, or never started before the global deadline.
4. Scheduling must prevent a whole category from being silently starved by earlier categories.
5. A failed build may serve a bounded last-good digest only with its original content time and explicit stale state.
6. Public coverage must be compact and safe. Raw host errors, exception strings, URLs, and precise failure timing remain operator telemetry.
7. `StoryPhase.FADING` must either have a validated meaning and working implementation or an explicit no-go result. Do not activate it only because the enum exists.
## Workstreams
- [ ] #7082 — P0: separate digest retention from forecast evidence and prune tombstones.
- [ ] #7083 — P0: make digest execution category-fair and classify deadline failures.
- [ ] #7084 — P1: serve bounded durable last-good digests with explicit stale state.
- [ ] #7085 — P1: expose compact digest coverage and latest-attempt state.
- [ ] #7081 — P2: validate and complete `StoryPhase.FADING` semantics.
Dependency order:
1. #7082 and #7083 can start immediately.
2. #7084 is blocked by #7083 because it consumes the attempt and failure vocabulary.
3. #7085 is blocked by #7083 and #7084 because it combines latest-attempt and stale-content semantics.
4. #7081 is independent and does not block the reliability frontier.
## Epic acceptance criteria
- [ ] The digest accumulator enforces its declared member-retention window in steady state. *(Corrected 2026-08-28: the shipped contract is **8 days**, not the planned 48 hours — `ACCUMULATOR_RETENTION_MS` in `scripts/_forecast-evidence-archive.mjs` is sized to the widest surviving reader, the ~7-day weekly digest lookback, plus a guard band. A 48-hour prune would truncate weekly digests. See the correction comment on #7082.)*
- [ ] The forecast resolver reads a dedicated, self-contained evidence archive that supports its declared 14-day maximum lookback.
- [ ] Migration and cleanup evidence records key scope, UTC time, member counts, oldest scores, and reader cutover. Cleanup does not start before forecast cutover is verified.
- [ ] A feed that never started is never reported as an upstream timeout.
- [ ] Every configured category has at least one scheduled opportunity before later rounds consume the remaining build budget, subject to the declared concurrency bound.
- [ ] A stale digest response identifies the original content time, latest attempt time, stale age, and stale reason.
- [ ] Stale content cannot silently drive alerts or automated decisions.
- [ ] The dashboard and MCP surface can distinguish complete, partial, stale, and unavailable news coverage without exposing raw operational telemetry.
- [ ] The lifecycle child either ships a tested FADING contract or records a tested no-go and corrects the promise without breaking the wire enum.
- [ ] Each child reaches ready-PR state separately. Merge and auto-merge remain separately authorized.
- [ ] The epic stays open until post-deploy evidence satisfies each applicable production gate. Green CI alone is not production acceptance.
## Out of scope
- Raising the five-item feed cap without an offline replay that proves distinct-publisher benefit inside the build budget.
- Adding broad variant or language support to all MCP NLP tools.
- Populating singular event locations from entity or country mentions.
- Adding historical `story:sources` values beside current-cycle `sourceCount`.
- Moving the digest builder to Railway before measured evidence shows that bounded Edge scheduling cannot meet the acceptance target.
- Increasing the current platform deadline to hide scheduling defects.
- Adding mutable production-health checks to pull-request acceptance.
## Required handoff
Every child must report these states separately: locally verified, PR ready, merged, deployed, production observed, and accepted. A later state must not be inferred from an earlier one.
Contributor guide
Research direction
Start with child issues #7082 and #7083, the frozen audit evidence, and scripts/_forecast-evidence-archive.mjs, including ACCUMULATOR_RETENTION_MS. This epic is not a self-contained first contribution: work is split across five dependent streams and requires production evidence. It is done only when the child acceptance criteria and applicable post-deploy gates are satisfied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- redis, typescript
- Domain
- api, backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100