forrtproject / forrtproject/flora-extractor
Stage 1: incremental pool updates — add new snapshot partitions without a full rescan or a full re-sync
- Dominant language
- Python
- Stars
- 2
- Forks
- 1
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 4
Description
## Why
The survivor pool is currently an all-or-nothing artifact. To pick up works OpenAlex has
published since the last scan, the only routes are a full 725 GB rescan (13–21 h) or a
`pool_sync --pull` of a pool someone else rescanned. Neither is proportionate to "a few
weeks of new works", and it means the pool ages between campaigns.
The machinery for incremental updates is *mostly already there*. This issue is about
closing the three specific gaps that stop it working, not about building it from scratch.
## What already works
Measured on this checkout against the live pool (2,232 partitions, 5,146,160 rows, 7.1 GB):
- **The scan already skips consumed partitions.** `_needs_scan()`
(`search/snapshot_scan.py`) returns False for any partition the ledger marks `done`
whose manifest `content_length` is unchanged. A scan against a refreshed manifest would
read only the new and rewritten partitions.
- **Writing a partition is already idempotent.** `_PoolWriter` names its output from the
partition URL (`_pool_file_name(url)`) and `commit()` does `tmp.replace(final)`, so
re-scanning a rewritten partition replaces its pool file rather than duplicating rows.
A partition that ends with no survivors deletes its old file.
- **The pull is already file-level incremental.** `pool_sync --pull` skips any file
already present on the other side with the same size, so fetching *added* partitions
costs only the added partitions.
## What blocks it
1. **A cached manifest is never refreshed.** `fetch_manifest(refresh: bool = False)`
returns `cache/snapshot/manifest.json` whenever it exists. `scan_snapshot()` calls it
as `fetch_manifest()`, and no CLI flag passes `refresh=True`. Once the manifest is
cached, a scan can never see a newer OpenAlex snapshot release — so the incremental
scan has nothing new to find.
2. **The ledger is not shared, so only the original scanning machine can scan
incrementally.** `pool_sync` moves the pool and the text overlay; it does not move
`cache/snapshot/ledger.json`. A collaborator who pulled the pool has 2,232 partitions
on disk and an empty `cache/snapshot/`, so `_needs_scan()` is True for every partition
and the "incremental" scan is a full 725 GB read. This is the state on at least one
current checkout.
3. **`expected_files` goes stale on append.** The sidecar's `expected_files` is the pool's
completeness check (`pool_fingerprint()` returns `None` below it). Appending partitions
makes the recorded value too low — harmless today because the check is `<`, but it
stops meaning "this pool is complete" the moment appends are routine.
## Proposal
- Expose the manifest refresh: `--refresh-manifest` on `run_search --scan`, or refresh
automatically when the cached manifest is older than N days. Report how many partitions
are new vs rewritten before reading any of them.
- Carry the ledger with the pool. Either sync it through `pool_sync` alongside
`_pool_provenance.json`, or reconstruct it from the pool: every pool file is named from
its partition URL, so `done` status is derivable from what is on disk — the only thing
missing is each partition's `content_length`, which the manifest supplies. The second
option needs no new transfer and fixes existing pulled pools.
- Re-stamp the sidecar at the end of an appending scan so `expected_files` matches what
the pool now holds.
- Decide what an appended pool means for release ids. `pool_fingerprint()` already hashes
every parquet's name, size and row count, so appending mints a new release id
automatically — which is correct, but it means every append re-routes the whole pool.
Whether that is acceptable, or whether routing should become incremental too, is the
open question and probably belongs with #144.
## Acceptance
- A second `--scan` on an up-to-date checkout reads 0 partitions and costs no bandwidth.
- A `--scan` after an OpenAlex release reads only the new and rewritten partitions, and
the resulting pool row count rises by exactly the survivors in those partitions.
- A checkout that obtained its pool via `pool_sync --pull` can do both of the above
without a full rescan.
- `pool_fingerprint()` returns a value (not `None`) on the appended pool.
## Notes
The gate must not change while doing this. Appending partitions scanned under a different
`search_gate_fingerprint` is already refused by both the ledger check and the sidecar
check in `scan_snapshot()`, and that refusal should stay exactly as strict — a pool that
is complete under neither gate is the failure mode those guards exist for.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading search/snapshot_scan.py, including fetch_manifest(), _needs_scan(), the ledger checks, and the sidecar handling, then trace run_search --scan and pool_sync. Verify the current behavior on an up-to-date checkout before changing manifest refresh, ledger transfer or reconstruction, and expected_files. Done means the listed acceptance cases pass without weakening the search gate checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100