koala73 / koala73/worldmonitor

fix(insights): repeated synthesis rejection preserves stale LKG while freshness metadata advances

Open
#5,947 4 comments 0 reactions 0 assignees View on GitHub
area: AI/intel area:seeders bug P2
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

## Summary

Production `seed-insights` repeatedly rejects the synthesized brief, preserves an older last-known-good payload, and still advances seed metadata. Compact health therefore reports no stale content even though the served insight payload has not advanced.

Priority: **P2**. Last-known-good preservation is the correct fail-safe, but attempted-run freshness must not masquerade as served-content freshness.

## Production evidence

Read-only Railway and Redis review on 2026-08-01:

- Every observed ten-minute cycle from approximately 06:51Z through 08:20Z logged:
- `[brief_synthesis] composer rejected output (parse/gates)`
- `No multi-source cluster available — publishing degraded (stories without brief)`
- `LKG preservation: existing payload is "ok", skipping degraded overwrite`
- The newest observed 08:20Z cycle failed in the same way.
- Canonical `news:insights:v1` still served the LKG generated at `2026-08-01T06:30:39.268Z`.
- That payload remained structurally healthy: status `ok`, 8 top stories, 14 multi-source clusters, and a 242-character world brief.
- `seed-meta:news:insights.fetchedAt` nevertheless advanced with failed attempts to roughly 08:20Z.
- Compact health reported no stale content, so the repeated synthesis failure and actual served-payload age were hidden.

The broad LLM migration plan in #4944 includes `seed-insights` telemetry, but it does not specifically own this production failure mode or the attempted-versus-served freshness contract.

## Impact

- Users and downstream agents keep receiving an aging LKG brief while health appears current.
- Repeated parser/gate rejection can continue indefinitely without a source-specific warning.
- Advancing only `fetchedAt` destroys the distinction between attempted execution, successful publication, and the age of served content.

## Scope

Diagnose the recurring composer parse/gate rejection, preserve LKG behavior, and make freshness truthful. Do not overwrite a valid LKG with degraded output simply to make timestamps move.

## Acceptance criteria

- [ ] Emit bounded rejection reason codes that distinguish parse failure, gate failure, missing cluster, and provider failure without logging prompt/output bodies.
- [ ] Record separate `lastAttemptAt`, `lastSuccessAt`, and served payload `generatedAt` semantics.
- [ ] Compact health becomes `WARNING` after a bounded number/age of consecutive failed syntheses even when an LKG is still served.
- [ ] A successful synthesis clears the warning only after a fresh payload is actually published.
- [ ] Preserve the LKG on failed/degraded attempts and retain its original source timestamp.
- [ ] Restore successful synthesis and verify a newly generated world brief across at least two consecutive production cycles.
- [ ] Tests cover repeated rejection with LKG, health transition, recovery, and no-LKG startup behavior.
- [ ] Coordinate telemetry/model work with #4944 rather than duplicating that migration.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the seed-insights synthesis and compact health paths, using the production evidence from Railway and Redis to compare attempted runs with the served LKG payload. Define the rejection reason, freshness, warning, recovery, and no-LKG cases from the acceptance criteria, then add tests for repeated rejection and recovery. Coordinate telemetry and model changes with #4944 and verify successful synthesis across two production cycles.

Written by the indexing model from the issue text.

Assessment

Tech stack
redis, typescript
Domain
ai, backend, observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.