Backup and cleanup wp1 selections
- Dominant language
- Python
- Stars
- 15
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
Recent incident on storage node proved that we should not get for granted that we will be able to regenerate wp1 selections should they get lost.
Should we backup these selections?
These are volumes of download.openzim.org:
```
2.5G ./archive
31G ./nightly
41G ./wp1
8.5G ./release
```
archive, nightly and release are already backed-up.
Note that wp1 volume is probably expected to grow, I don't find anything to clean this up. But maybe this is just another missing part, pretty easy to fix as well.
That been said, if we look at a given folder content `enwiki_2025-01`, we notice that maybe we do not really need all this:
```
5.1G pagelinks.tsv.zip
386M ./projects
367M langlinks.tsv.zip
247M pages.tsv.zip
222M all.tsv.zip
210M ratings.tsv.zip
147M projects.zip
103M pageviews.tsv.zip
80M redirects.tsv.zip
64M scores.tsv.zip
32M ./tops
24M ./customs
15M tops.zip
9.5M customs.zip
66K vital.tsv.zip
625 README
```
Do we really need these zips which represent the sheer volume, especially pagelinks.tsv.zip? AFAIK, we do not use them on the Zimfarm. And nobody raised an issue these were missing during the downtime.
@kelson42 this is for you to provide background / decisions
Contributor guide
Research direction
Start by comparing the wp1 volume with the already-backed-up archive, nightly, and release volumes, then review the enwiki_2025-01 contents listed in the issue. Confirm which selections and archives are still needed by Zimfarm before proposing backup or cleanup work. Done means the retention decision and operational change are documented and verified.
Written by the indexing model from the issue text.
Assessment
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100