koala73 / koala73/worldmonitor
fix(news): separate digest retention from forecast evidence and prune tombstones
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
## Parent
#7080
## Priority
P0 — the current structure is unbounded and shares one key between incompatible 48-hour and 7–14-day contracts.
## What to build
Give digest accumulation and forecast judging separate retention contracts. The digest accumulator must stay bounded to 48 hours. Forecast judging must use a dedicated, self-contained archive that remains usable for its declared maximum lookback and does not depend on seven-day story-track rows.
## Why
The current writer refreshes the accumulator key TTL on every build but never removes old sorted-set members. Production contains millions of expired members and tombstones. A direct 48-hour trim is unsafe because the forecast resolver reads the same key for up to 14 days.
## Implementation plan
### 1. Freeze reader and data contracts
- Inventory every in-repo reader of digest accumulators and record its key scope, lookback, fields, and failure behavior.
- Define the digest contract as a 48-hour rolling index used by briefs, watchlists, and spike calculations that request no longer history.
- Define the forecast archive as a versioned, self-contained evidence record with the minimum fields required to judge a forecast. Its maximum retention must cover the 14-day reader contract plus a small cleanup guard band.
- Do not make the forecast archive depend on a story hash whose record expires after seven days.
### 2. Add the archive before pruning
- Add dual publication of eligible full-English evidence to the new forecast archive.
- Backfill the still-readable portion of the last 14 days before any destructive cleanup. Record unreadable tombstones and missing fields instead of silently omitting them.
- Keep archive writes bounded by explicit record, byte, and command budgets.
- Add archive version, publication time, source digest time, and retention metadata.
### 3. Migrate and verify the forecast reader
- Move forecast judging to the dedicated archive.
- Compare old and new reader results over the same eligible window while both paths exist.
- Fail closed when archive coverage is truncated, unreadable, or outside the declared window. Do not convert missing evidence into a judged negative.
- Remove the old accumulator dependency only after the comparison passes.
### 4. Enforce digest retention
- Add member-level pruning for scores older than 48 hours as part of normal digest publication.
- Keep key TTL as abandoned-key cleanup, not as member retention.
- Add a bounded cleanup tool with dry-run output. It must enumerate exact keys and report cardinality and oldest/newest score before mutation.
- Run the one-time production sweep only after the forecast cutover is deployed and verified. Capture the pre-sweep evidence needed for recovery or audit.
### 5. Observe steady state
- Record accumulator cardinality, oldest age, prune count, and command latency in operator monitoring.
- Keep ordinary cardinality drift out of GitHub status checks. Integrity, malformed archive records, and retention-contract violations must still fail closed where they affect judging.
## Acceptance criteria
- [ ] All accumulator readers and their required lookbacks are listed in the pull-request handoff.
- [ ] Forecast judging uses a dedicated archive and no longer reads a digest accumulator.
- [ ] A forecast evidence item remains readable at day 13 and is ineligible after the declared maximum retention.
- [ ] The archive remains usable after the corresponding story-track row expires.
- [ ] A digest accumulator member older than 48 hours is removed during normal publication.
- [ ] Boundary tests keep exactly eligible members at the 48-hour edge and do not remove newer members.
- [ ] The migration reports old/new reader parity, archive truncation, missing evidence, and tombstones.
- [ ] The cleanup tool defaults to dry-run and requires explicit key scope before mutation.
- [ ] The production sweep records UTC time, key count, member counts, oldest ages, and the remaining forecast-archive coverage.
- [ ] Post-deploy observation shows bounded cardinality for at least three consecutive digest publication cycles.
## Test plan
- Use a fake clock to test 48-hour digest and 14-day forecast boundaries.
- Test archive reads after story-track expiry.
- Test malformed, truncated, duplicate, and partially backfilled archive records.
- Test that a failed archive write does not advance coverage metadata.
- Test the cleanup dry-run, exact-key allowlist, and idempotent second run.
- Run focused news, forecast-resolution, Redis-cache, and story-tracking tests, followed by the API typecheck and diff hygiene checks.
## Blocked by
None — can start immediately.
## Out of scope
- Changing forecast scoring or judgment policy.
- Extending digest retention beyond 48 hours.
- Treating tombstones as negative forecast evidence.
- Pruning production before the new reader is deployed and verified.
- Hiding ordinary storage health inside pull-request CI.
Contributor guide
Research direction
Start by tracing digest publication and forecast-resolution readers, then run the focused news, forecast-resolution, Redis-cache, and story-tracking tests to inventory lookbacks and failure behavior. Done means forecast judging uses the dedicated archive, digest members are bounded at 48 hours, boundary and malformed-record tests pass, and the cleanup remains dry-run by default.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- redis, typescript
- Domain
- backend, databases, observability
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100