koala73 / koala73/worldmonitor

fix(news): separate digest retention from forecast evidence and prune tombstones

Open
#7,082 7 comments 0 reactions 0 assignees View on GitHub
agent-readiness area: AI/intel area:news area:seeders bug High Value P0
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

## Parent

#7080

## Priority

P0 — the current structure is unbounded and shares one key between incompatible 48-hour and 7–14-day contracts.

## What to build

Give digest accumulation and forecast judging separate retention contracts. The digest accumulator must stay bounded to 48 hours. Forecast judging must use a dedicated, self-contained archive that remains usable for its declared maximum lookback and does not depend on seven-day story-track rows.

## Why

The current writer refreshes the accumulator key TTL on every build but never removes old sorted-set members. Production contains millions of expired members and tombstones. A direct 48-hour trim is unsafe because the forecast resolver reads the same key for up to 14 days.

## Implementation plan

### 1. Freeze reader and data contracts

- Inventory every in-repo reader of digest accumulators and record its key scope, lookback, fields, and failure behavior.
- Define the digest contract as a 48-hour rolling index used by briefs, watchlists, and spike calculations that request no longer history.
- Define the forecast archive as a versioned, self-contained evidence record with the minimum fields required to judge a forecast. Its maximum retention must cover the 14-day reader contract plus a small cleanup guard band.
- Do not make the forecast archive depend on a story hash whose record expires after seven days.

### 2. Add the archive before pruning

- Add dual publication of eligible full-English evidence to the new forecast archive.
- Backfill the still-readable portion of the last 14 days before any destructive cleanup. Record unreadable tombstones and missing fields instead of silently omitting them.
- Keep archive writes bounded by explicit record, byte, and command budgets.
- Add archive version, publication time, source digest time, and retention metadata.

### 3. Migrate and verify the forecast reader

- Move forecast judging to the dedicated archive.
- Compare old and new reader results over the same eligible window while both paths exist.
- Fail closed when archive coverage is truncated, unreadable, or outside the declared window. Do not convert missing evidence into a judged negative.
- Remove the old accumulator dependency only after the comparison passes.

### 4. Enforce digest retention

- Add member-level pruning for scores older than 48 hours as part of normal digest publication.
- Keep key TTL as abandoned-key cleanup, not as member retention.
- Add a bounded cleanup tool with dry-run output. It must enumerate exact keys and report cardinality and oldest/newest score before mutation.
- Run the one-time production sweep only after the forecast cutover is deployed and verified. Capture the pre-sweep evidence needed for recovery or audit.

### 5. Observe steady state

- Record accumulator cardinality, oldest age, prune count, and command latency in operator monitoring.
- Keep ordinary cardinality drift out of GitHub status checks. Integrity, malformed archive records, and retention-contract violations must still fail closed where they affect judging.

## Acceptance criteria

- [ ] All accumulator readers and their required lookbacks are listed in the pull-request handoff.
- [ ] Forecast judging uses a dedicated archive and no longer reads a digest accumulator.
- [ ] A forecast evidence item remains readable at day 13 and is ineligible after the declared maximum retention.
- [ ] The archive remains usable after the corresponding story-track row expires.
- [ ] A digest accumulator member older than 48 hours is removed during normal publication.
- [ ] Boundary tests keep exactly eligible members at the 48-hour edge and do not remove newer members.
- [ ] The migration reports old/new reader parity, archive truncation, missing evidence, and tombstones.
- [ ] The cleanup tool defaults to dry-run and requires explicit key scope before mutation.
- [ ] The production sweep records UTC time, key count, member counts, oldest ages, and the remaining forecast-archive coverage.
- [ ] Post-deploy observation shows bounded cardinality for at least three consecutive digest publication cycles.

## Test plan

- Use a fake clock to test 48-hour digest and 14-day forecast boundaries.
- Test archive reads after story-track expiry.
- Test malformed, truncated, duplicate, and partially backfilled archive records.
- Test that a failed archive write does not advance coverage metadata.
- Test the cleanup dry-run, exact-key allowlist, and idempotent second run.
- Run focused news, forecast-resolution, Redis-cache, and story-tracking tests, followed by the API typecheck and diff hygiene checks.

## Blocked by

None — can start immediately.

## Out of scope

- Changing forecast scoring or judgment policy.
- Extending digest retention beyond 48 hours.
- Treating tombstones as negative forecast evidence.
- Pruning production before the new reader is deployed and verified.
- Hiding ordinary storage health inside pull-request CI.

Contributor guide

Open the contributing guide

Research direction

Start by tracing digest publication and forecast-resolution readers, then run the focused news, forecast-resolution, Redis-cache, and story-tracking tests to inventory lookbacks and failure behavior. Done means forecast judging uses the dedicated archive, digest members are bounded at 48 hours, boundary and malformed-record tests pass, and the cleanup remains dry-run by default.

Written by the indexing model from the issue text.

Assessment

Tech stack
redis, typescript
Domain
backend, databases, observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.