forrtproject / forrtproject/flora-extractor

Archive reject verdicts to HF so Postgres holds only what validation still needs

Open
#153 0 comments 0 reactions 0 assignees View on GitHub
stage-2
Dominant language
Python
Stars
2
Forks
1
Avg merge
1d 6h
Merged PRs (30d)
4

Description

Follow-up to #146 (M2). Not urgent: the trigger is the first campaign large enough to
push the Supabase database past its plan limit.

## The problem, measured

`filter/engine/sizing.py` (the §8-1 measurement) puts `engine_verdicts` at **542.5 B/row**
against `engine_claim_items` at 130.4 B/row. Verdicts are ~89% of the footprint, and the
two-voter tiers write two of them per work:

| campaign | claim_items | verdicts | total | 500 MB free tier |
| --- | ---: | ---: | ---: | --- |
| 300k works × 2 verdicts | 37.3 MB | 310.4 MB | 347.8 MB | fits |
| 800k works × 2 verdicts | 99.5 MB | 827.7 MB | 927.6 MB | exceeded |

So a full-residue campaign forces a plan upgrade, and the term forcing it is verdict
volume rather than anything the transactional store actually needs online.

## Why this is not a "move off Postgres" problem

Worth writing down so it is not relitigated. The database cost is noise next to the work
it supports: the 2026-08-03 pilot screened 3,000 rows for $5.40 (~$0.0018/row), so the
same 800k-row campaign spends roughly **$1,400 on LLM calls** against $25/month for a
Supabase plan. Re-architecting to a pay-per-use key-value store would also break the
property the whole claim design rests on — DynamoDB's `TransactWriteItems` caps at 100
items, so a batch claim would become N transactions plus compensating cleanup, which is
exactly the select-then-insert race #146 §4 forbids — and it would split the mutable
authority away from the validation tables that already live in Supabase.

The right fix is the storage split #146 §4 already specifies: **HF holds bulk immutable
data; Postgres holds what is live and mutable.** A settled verdict on a rejected work is
bulk immutable data sitting in the wrong home.

## Scope: archive rejects, retain progressions

Only verdicts that **reject** a work are archived:

- `screen_cheap` two-explicit-noes discards;
- `screen_expensive` verdicts whose gate outcome was `discard` / `not_a_replication`.

Verdicts that let a work **progress toward validation are retained in Postgres**, because
they are the live lineage behind a record that Stage 3 resolves and a human validates —
that chain has to be joinable online, and #146 §5 supersession keys on it. Rejects have no
such downstream: nothing links to them, and they are the majority of rows (the pilot's
article arms proceeded at ~12.5%), so archiving them recovers most of the space while
leaving every record a validator could ever ask about in the database.

## Sketch

1. A campaign closes (all its claims `complete`/`cancelled`/`failed`).
2. Reject verdicts for that campaign are written to HF as parquet, in a frozen immutable
manifest — the same release/manifest machinery `pool_sync` and the M3 overlays use, one
identity scheme, not a third.
3. Only after the upload is confirmed, the archived rows are deleted from Postgres and
replaced by one pointer row per campaign (archive manifest hash, row count, date range).
This is the single place the permanence triggers must permit a delete, so it needs an
explicit, narrow, audited path — a verdict may leave the database only into an archive
that is already durable, never into nothing.
4. `engine_audit` records the archival; a reader wanting an archived verdict resolves the
pointer to the HF manifest.

Evidence stays permanent throughout — it changes address, not existence. The raw responses
are already on HF by `response_hash`, so this moves the lean row to sit beside its own
evidence.

## Open questions

- Whether the pointer is per campaign or per claim (per campaign is fewer rows; per claim
is easier to reason about when a campaign is re-run).
- Whether a superseded reject (`superseded_by` set) is archivable, or must stay online
until its superseding verdict also settles.
- Retention: do archived rejects ever expire, or is HF the permanent home? #146 §4's
redaction path (open decision 3) interacts here.

Contributor guide

No contributing guide indexed for this repository

Research direction

Read #146 §4 and the existing pool_sync and M3 overlay release-manifest machinery first; inspect filter/engine/sizing.py and engine_audit for the current storage and audit boundaries. Before implementation, resolve the pointer, supersession, and retention questions; done means an agreed, narrowly audited campaign-close flow that preserves evidence and supports archived-verdict lookup.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, postgres, python
Domain
backend, data, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.