forrtproject / forrtproject/flora-extractor
Stage 1: curated harvester for OSF/I4R/HAL/EconStor reproduction reports
- Dominant language
- Python
- Stars
- 2
- Forks
- 1
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 4
Description
## Why
FLoRA holds almost only replications. Auditing the reproduction entry sheet (93 distinct reports;
`analysis/stage_b_eval/reproduction_coverage_report.md` on `analysis/stage-b-eval`) shows the
vocabulary side is already solved and the gap is structural:
- **84/93 (90%) are OpenAlex-indexed**, and of those the pipeline already admits **98.8%** at
Stage 1. DOI presence predicts indexing almost perfectly: 97.7% of rows with a `doi_r` are
indexed, **0%** of rows without one.
- All 22 measured reproduction-vocabulary candidates recover ≈nothing not already caught. I4R
boilerplate never appears in an OpenAlex title or abstract, and OpenAlex has no I4R source
record — so no keyword work reaches these.
- **9/93 (9.7%) are not in OpenAlex at all**: 5 OSF-hosted, 1 HAL, 1 EconStor handle mis-written
as a DOI, 1 OSF DOI, 2 bare-DOI-string rows. Every one carries clear reproduction vocabulary and
**would be admitted if indexed** (8 positive, 1 ambiguous).
## What is needed
A Stage 1 harvester pulling OSF / I4R / HAL / EconStor reproduction reports into `candidates.csv`
with a curated source tag.
Two things the existing bypass does *not* give for free:
- `CURATED_SOURCES` is a set of **exact source strings**, currently `i4r`, `bob_reed`,
`backfill_old_pipeline`. A harvester emitting `osf`, `hal` or `econstor` bypasses nothing until
those tags are added (or a generic curated-provenance flag replaces the string set).
- The Stage 2 curated bypass runs **after** `non_article_doi()`, so a curated repository object can
still be rejected before the bypass is consulted. Needs a test.
Scope note: the bypass covers Stage 2 and the cheap pre-screen only — curated rows still go
through the validated Stage 3 screen, which is the desired behaviour and should be stated
explicitly so nobody plumbs an end-to-end bypass by mistake.
Also needed: identifier policy and dedup precedence for DOI-less reports (OSF URLs), so they get a
stable key in the work/alias table #146 introduces.
Related: #144 (Wave R), #146, #148.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with analysis/stage_b_eval/reproduction_coverage_report.md and trace the existing CURATED_SOURCES and non_article_doi paths before designing the OSF, I4R, HAL, and EconStor harvest. Define the identifier and deduplication behavior for DOI-less OSF URLs, add coverage for curated repository objects rejected before the Stage 2 bypass, and verify that harvested rows reach candidates.csv while still passing through Stage 3.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100