forrtproject / forrtproject/flora-extractor

study_r / study_o must be study identifiers everywhere, never paper titles

Open
#103 1 comment 0 reactions 0 assignees View on GitHub
stage-3
Dominant language
Python
Stars
2
Forks
1
Avg merge
1d 6h
Merged PRs (30d)
4

Description

FLoRA's codebook defines `study_o` as the **target study number inside the original paper** ("Study number, can be multiple", e.g. `1, 2`), coded from the replication's abstract. Across this repo and the validation DB the same names are used for **paper titles**. One name, two meanings — and the real study numbers had nowhere to go.

`extracted.csv` now has a real `study_o` (numbers), and `extract/csv_to_db.py` sends titles as `title_r` / `title_o` and `study_o` as the number (PR: `audit/pr10-flora-rule-alignment`). The rest is listed here.

### Blocking — validation repo (`flora-validation`)

`csv_to_db` will fail on every insert until the `unvalidated` table has these columns. Nothing is written and nothing is corrupted in the meantime, but **the import is unusable until this lands**.

- [ ] `ALTER TABLE unvalidated ADD COLUMN title_r text, ADD COLUMN title_o text;`
- [ ] Backfill `title_r` / `title_o` from the existing `study_r` / `study_o` values (they currently hold titles).
- [ ] Clear `study_r` / `study_o` after the backfill, then let `csv_to_db` repopulate `study_o` with study numbers. `study_r` stays empty — see below.
- [ ] Update the validator UI to read the title from `title_r` / `title_o` and to show `study_o` as "Study 2 of the original", not as a title.
- [ ] `db_schema.sql` comments on both columns saying they are identifiers.

### This repo — legacy readers that treat `study_r` as a title

These read seeded CSVs (`all_replications.csv`, `flora_selected.csv`, old-pipeline exports) whose `study_r` column really does hold a title, so they are not wrong today — they are the reason the confusion spread. They need renaming to `title_r` at the same time as the CSVs they read.

- [ ] `validate/routes/batch.py:74,84,312`
- [ ] `validate/routes/multi_originals.py:35,61`
- [ ] `validate/routes/dashboard.py:555,581,585`
- [ ] `validate/templates/batch.html:314,381`
- [ ] `validate/templates/multi_originals.html:149,215`
- [ ] `validate/templates/dashboard.html:3216,3246`
- [ ] `validate/static/render.js:92,95,169,272` — line 95 falls back to `data.study_o` for a title, which is exactly the conflation
- [ ] `analysis/old_pipeline_compare.py:182,215`
- [ ] `extract/run_extract.py:_merge_row` — `title_r` falls back to `row["study_r"]` for old seeded data; drop once the seeds are renamed

### Open question — is there a `study_r` at all?

FLoRA's codebook has no `study_r`; the `study_o` row explicitly notes "Not study_r". The validation DB has the column. Either it means "which study of the replication paper" (which the pipeline does not code, so it would stay empty) or it should be dropped. Worth settling before the backfill above.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with validation/db_schema.sql, validation's unvalidated table, and extract/csv_to_db.py to confirm the intended column meanings. Then trace the listed batch, multi-originals, dashboard, template, render, and legacy pipeline readers, including the study_o title fallback. Done means the schema, backfill, UI labels, seeded CSV readers, and imports consistently keep titles in title_r/title_o and study_o as a number, after deciding the fate of study_r.

Written by the indexing model from the issue text.

Assessment

Tech stack
html, javascript, python, sql
Domain
backend, databases, frontend
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.