forrtproject / forrtproject/flora-extractor

Stage 1: incremental pool updates — add new snapshot partitions without a full rescan or a full re-sync

Open
#203 0 comments 0 reactions 0 assignees View on GitHub
enhancement stage-1
Dominant language
Python
Stars
2
Forks
1
Avg merge
1d 6h
Merged PRs (30d)
4

Description

## Why

The survivor pool is currently an all-or-nothing artifact. To pick up works OpenAlex has
published since the last scan, the only routes are a full 725 GB rescan (13–21 h) or a
`pool_sync --pull` of a pool someone else rescanned. Neither is proportionate to "a few
weeks of new works", and it means the pool ages between campaigns.

The machinery for incremental updates is *mostly already there*. This issue is about
closing the three specific gaps that stop it working, not about building it from scratch.

## What already works

Measured on this checkout against the live pool (2,232 partitions, 5,146,160 rows, 7.1 GB):

- **The scan already skips consumed partitions.** `_needs_scan()`
(`search/snapshot_scan.py`) returns False for any partition the ledger marks `done`
whose manifest `content_length` is unchanged. A scan against a refreshed manifest would
read only the new and rewritten partitions.
- **Writing a partition is already idempotent.** `_PoolWriter` names its output from the
partition URL (`_pool_file_name(url)`) and `commit()` does `tmp.replace(final)`, so
re-scanning a rewritten partition replaces its pool file rather than duplicating rows.
A partition that ends with no survivors deletes its old file.
- **The pull is already file-level incremental.** `pool_sync --pull` skips any file
already present on the other side with the same size, so fetching *added* partitions
costs only the added partitions.

## What blocks it

1. **A cached manifest is never refreshed.** `fetch_manifest(refresh: bool = False)`
returns `cache/snapshot/manifest.json` whenever it exists. `scan_snapshot()` calls it
as `fetch_manifest()`, and no CLI flag passes `refresh=True`. Once the manifest is
cached, a scan can never see a newer OpenAlex snapshot release — so the incremental
scan has nothing new to find.

2. **The ledger is not shared, so only the original scanning machine can scan
incrementally.** `pool_sync` moves the pool and the text overlay; it does not move
`cache/snapshot/ledger.json`. A collaborator who pulled the pool has 2,232 partitions
on disk and an empty `cache/snapshot/`, so `_needs_scan()` is True for every partition
and the "incremental" scan is a full 725 GB read. This is the state on at least one
current checkout.

3. **`expected_files` goes stale on append.** The sidecar's `expected_files` is the pool's
completeness check (`pool_fingerprint()` returns `None` below it). Appending partitions
makes the recorded value too low — harmless today because the check is `<`, but it
stops meaning "this pool is complete" the moment appends are routine.

## Proposal

- Expose the manifest refresh: `--refresh-manifest` on `run_search --scan`, or refresh
automatically when the cached manifest is older than N days. Report how many partitions
are new vs rewritten before reading any of them.
- Carry the ledger with the pool. Either sync it through `pool_sync` alongside
`_pool_provenance.json`, or reconstruct it from the pool: every pool file is named from
its partition URL, so `done` status is derivable from what is on disk — the only thing
missing is each partition's `content_length`, which the manifest supplies. The second
option needs no new transfer and fixes existing pulled pools.
- Re-stamp the sidecar at the end of an appending scan so `expected_files` matches what
the pool now holds.
- Decide what an appended pool means for release ids. `pool_fingerprint()` already hashes
every parquet's name, size and row count, so appending mints a new release id
automatically — which is correct, but it means every append re-routes the whole pool.
Whether that is acceptable, or whether routing should become incremental too, is the
open question and probably belongs with #144.

## Acceptance

- A second `--scan` on an up-to-date checkout reads 0 partitions and costs no bandwidth.
- A `--scan` after an OpenAlex release reads only the new and rewritten partitions, and
the resulting pool row count rises by exactly the survivors in those partitions.
- A checkout that obtained its pool via `pool_sync --pull` can do both of the above
without a full rescan.
- `pool_fingerprint()` returns a value (not `None`) on the appended pool.

## Notes

The gate must not change while doing this. Appending partitions scanned under a different
`search_gate_fingerprint` is already refused by both the ledger check and the sidecar
check in `scan_snapshot()`, and that refusal should stay exactly as strict — a pool that
is complete under neither gate is the failure mode those guards exist for.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading search/snapshot_scan.py, including fetch_manifest(), _needs_scan(), the ledger checks, and the sidecar handling, then trace run_search --scan and pool_sync. Verify the current behavior on an up-to-date checkout before changing manifest refresh, ledger transfer or reconstruction, and expected_files. Done means the listed acceptance cases pass without weakening the search gate checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.