boostorg / boostorg/website-v2
Task: Rework the formal-review results importer for the new boost.org page
- Dominant language
- HTML
- Stars
- 18
- Forks
- 28
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 81
Description
> **Implements the outcome of spike [#2485](https://github.com/boostorg/website-v2/issues/2485)**,
## Context
`import_reviews` scrapes `https://www.boost.org/community/review_schedule.html`, **which no longer
exists**, and matches tables by `summary="Review Results"`, an attribute the replacement page does
not have. The command therefore finds nothing and returns quietly.
The results now come from a regularly published Antora document,
`boostorg/website-v2-docs/formal-reviews/.../review-results.adoc`, served at
`https://www.boost.org/doc/formal-reviews/review-results.html`. Three things about that page break
the old approach beyond the URL:
1. The content is HTML-escaped inside an `` wrapper.
2. The section now **opens with a year-navigation table**, and the old code took the first table
after the heading. Pointing the existing command at the new URL would import roughly two junk
rows instead of the 286 real reviews - and report success.
3. boost.org serves the page as `text/html` with no charset, so `requests` falls back to
ISO-8859-1 per the HTTP spec and every accented reviewer name is stored mojibaked
(`Joaquín M López Muñoz` becomes `JoaquÃn M López Muñoz`).
Separately, the command has always created near-duplicate rows on every run, because its
`update_or_create` key is too narrow for the data.
This matters beyond the reviews page itself: `versions.models.Review` is the intended source for
the **Reviewer** badge. Without a working importer that badge counts nothing.
## Scope
1. Point the importer at the current URL, and extract the real document from the content iframe.
2. Force UTF-8 decoding.
3. **Parse every result table in the section, not the first.** Match tables by their header tuple
and walk until the next `h2`, so the navigation table is rejected and each year's table is
picked up.
4. Replace the narrow `update_or_create` key with a normalized fingerprint (submission, submitter,
review dates - accent-stripped, lowercased, punctuation removed) so a re-import updates in
place, pre-existing duplicates collapse, and a library reviewed twice on different dates stays
two records.
5. Import oldest-first so the newest review gets the highest id, matching the `-id` ordering the
admin and the public page use.
6. Handle superseded results via the current markup (`<span class="line-through">`, not `<del>`).
7. **Fail loudly.** An unparseable page must raise, not return quietly - this will also run from a
scheduled task where nobody is watching the output.
8. Time out the fetch, so a boost.org that accepts the connection and never answers cannot hold a
worker for the life of the process.
9. Move `--clean` inside the import transaction and after the parse, so a failed scrape cannot
leave the reviews deleted.
10. Call `discard_source_achievements` before deleting a duplicate review. `UserAchievement`
reaches its source through a generic FK with no referential integrity, so a bare delete leaves
a grant still counting toward a threshold.
11. Only store GitHub URLs as `github_link`, matched on hostname. Omit the key otherwise so a
re-import cannot erase a link entered by hand in the admin.
12. `ReviewAdmin`: id, review-manager and scraped-review-manager columns, newest-first ordering,
wider search, and the `select_related` / `prefetch_related` the results column needs.
13. `import_reviews_task`, so the scrape can run off-request.
## Out of scope
- Importing upcoming / scheduled reviews. The page still lists them; nothing consumes them.
- Any change to `versions.models.Review` - the fields all exist already.
- The `library-review` achievement source itself. Separate ticket.
- The admin button that starts the task. Separate ticket (the task-button infrastructure).
- `documentation_link`. The submission cell also carries project pages and announcement posts, and
guessing which of them is documentation is not this command's job.
## Acceptance criteria
- [ ] `import_reviews` populates `Review` and `ReviewResult` from the current page
- [ ] All result tables are parsed and the navigation table is rejected, verified against the live
page (expect ~286 rows, 0 without a result)
- [ ] Accented names round-trip correctly into both the raw fields and the FK lookup
- [ ] A second run creates 0 reviews and deletes 0 rows
- [ ] Pre-existing duplicates collapse, and the count is reported
- [ ] Deleting a duplicate discards the achievements sourced from it
- [ ] An unparseable page raises `CommandError`
- [ ] `--clean` cannot leave the table empty after a failed scrape
- [ ] Superseded results are marked `is_most_recent=False`
- [ ] Reviews appear newest-first in the admin and on the public page
- [ ] Parser is tested against a captured sample page
- [ ] Full suite green, pre-commit clean
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.