openzim / openzim/python-scraperlib
Integrate KiwixStorage
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31
- Forks
- 27
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 2
Description
Ours scrapers all depend on S3 and use KiwixStorage for it. That wrapper and repo are mostly untouched (yet working) and would great benefit from being integrated into scraperlib: tests, usage, dependencies update, better API, etc.
Now that we had experience using it, it probably make sense to reduce its API to what's actually used.
It's true that some non-scraper projects are using it (cardshop and zimfarm to upload to S3), but those are also using scraperlib and we've discussed making scraperlib's dependencies more modular via extras anyway.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing how scraperlib and its scrapers currently use S3 and KiwixStorage, then compare that usage with the python-storagelib API. Done means adding the requested tests and integration, updating dependencies, and defining the smaller API and modular dependency structure described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend-api-design, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100