Scrapers: test assumptions over the whole corpus once in a while

In scrapers, we implement logic based on some knowledge and rules of the upstream data.

For instance, in Gutenberg scrapers we have a logic to remove some data (license, ...) from ePub and HTML files. Due to upstream content when the scraper has been written, we've created a logic capable to handle various formats of the data to be remove and of what needs to be left in the ZIM.

We have few tests checking this logic still behaves as intended, but this is focused at checking that we do not break the functionality during code changes.

Should upstream change (and it does), then we end up with two issues.

First, the ZIM will containing unwanted or broken content. This will usually be detected very late in the chain (end-users complaining) but will ultimately be fixed.

Second problem is more pernicious. Since we generally can't know if the bug is present on the whole corpus (e.g. Gutenberg books) or only the one the issue has been raised and few ones we can manually check for, we generally keep the old logic and add new logic on-top or side-by-side to fix the bug.

https://github.com/openzim/gutenberg/issues/437 is another good example: we can quite easily fix the issue but we will have to keep old logic "should this be working differently somewhere".

And we obviously have the same kind of problem in all scrapers: sotoki where we assume content is in HTML escaped or in Markdown, where we assume content is already sorted or not, ...

I feel like we should find a solution to this problem. I would propose we imagine something based on regular (once a month, year, on-demand, ...) checks on the whole corpus. This probably means code changes to be able to report which logic is really triggered, and capabilities to run that on the whole corpus.

I know this is a general issue in IT in general, but since we are lucky enough to have sufficiently small datasets that we can run checks on them + we have full access to these datasets (no confidentiality issue, ...), I wanted to bring light on this topic.

The issue names no files, tests, or entry points. Start by mapping the scraper logic for Gutenberg and sotoki, then define how whole-corpus checks should run and report which assumptions are triggered. Done means the checks can run regularly or on demand and expose upstream changes across the corpus.