Full-text acquisition: five routes measured to work that the waterfall does not try

Follow-up to the acquisition fixes of 2026-08-08 (commit a0ff034: the `row_url` tier, the two request identities, Word documents, the OSF preprint pattern).

Thirty works that Stage 3 acquired nothing for were investigated by hand against live servers. **25 of the 30 turned out to be obtainable**, 2 were genuinely paywalled, 1 was a dead link, and 2 were code/data deposits with no manuscript to code. The waterfall got essentially none of them.

Two of the routes found — DataCite metadata, and walking every OA location rather than the first — are basic enough that they are being implemented directly rather than tracked here. The five below are the remainder.

## 1. Crossref bibliographic title search → OA or preprint copy

The highest value per unit cost of anything found, and the only route that helps a row with **no DOI at all**.

https://api.crossref.org/works?rows=5&query.bibliographic=

Take any `10.31234/*` (PsyArXiv) or open-access container from the results, then read `citation_pdf_url` off its landing page. It beat Unpaywall on every row where both were tried, because Unpaywall knew only the publisher copy and the publisher was Cloudflare-walled.

It also repairs dead links: CiteSeerX is dead corpus-wide and several Ghent `biblio` ids in this data are stale, and in both cases the title search recovered a live DOI.

Free. Fixed 4 of the 30.

## 2. Legacy DSpace REST and OAI-PMH, which sit outside the bot walls

On EconStor the HTML site, the `/bitstream/` path and the DSpace 7 `/server/api` path were all walled, while these two answered plain curl:

https://<host>/rest/handle/<prefix>/<id>?expand=bitstreams -> retrieveLink
https://<host>/oai/request?verb=GetRecord&metadataPrefix=didl&identifier=oai:<host>:<prefix>/<id>

Generalises across the EconStor/RePEc economics segment, which is a substantial share of this corpus's repository rows. Free, no browser needed.

## 3. Unpaywall green locations must be scraped, not read

When a location has `url_for_pdf: null` but `host_type: "repository"`, the PDF is on the landing page, not in the API response. Fetch `url_for_landing_page`, take the on-page `/bitstream/handle/...?sequence=1&isAllowed=y` href — not the `citation_pdf_url` meta value, which 403'd — and send a `Referer` header. One extra request; these rows are currently dropped.

## 4. EPrints and Figshare landing patterns

- EPrints: landing page → the single `href` under `/<id>/<n>/*.pdf`, or the `citation_pdf_url` meta tag.
- Figshare: `https://api.figshare.com/v2/articles/<id>` where `<id>` is the numeric tail of the DataCite DOI → `files[].download_url`. Covers every Figshare-backed institutional repository.

## 5. A headless-browser document tier for JS-challenge hosts

Some repositories serve the file only after a JavaScript challenge completes — Anubis proof-of-work, AWS WAF. A real browser engine passes these by doing the work the page asks for; no CAPTCHA is solved and no credential is used. This is the only route to a whole class of DSpace 7 and Figshare repositories.

Costs ~15-30 s and ~200 MB RAM per document, so it belongs at the bottom of the waterfall, below every HTTP route.

Note for whoever implements it: the existing `playwright` tier is the natural home, but it must NOT present a browser User-Agent to Anubis-protected hosts, which is what caused the failures fixed in a0ff034.

## Two things worth knowing before implementing any of this

**A server can return the wrong paper for the right URL.** Two fetches of one PLOS printable URL returned two different papers. A document-identity check is being added separately; do not add a route without it.

**Some rows are not fixable by any route**, and the correct outcome is to record that: a Zenodo deposit of R code with no manuscript, a conference abstract with no full text in existence, a master's thesis whose file was never deposited. There are also corpus metadata errors — dead `url_r` values and at least one `journal_r` naming the wrong university.

## Measured summary of the 30

| outcome | n |
| ------- | -: |
| obtainable by a route above | 25 |
| genuinely paywalled (institutional access is the answer) | 2 |
| dead link, file never deposited | 1 |
| data/code deposit, no manuscript exists | 2 |

Start by reading the existing acquisition waterfall and its Playwright tier, then review the separate document-identity check before adding any route. Implement and validate the Crossref, DSpace/OAI-PMH, Unpaywall, EPrints/Figshare, and browser routes in their stated order, with the browser tier last; done means the measured repository patterns are covered without accepting the wrong paper.

Written by the indexing model from the issue text.