openzim / openzim/python-scraperlib
Add S3 based optimization cache support
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31
- Forks
- 27
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 2
Description
We use kiwix_storagelib for implementing S3 based optimization cache in the scrapers. However, this gives rise to redundant code. We put a version of the file along with the optimizer version as the metadata always. So, this can be better implemented in scraperlib. For a start, we can have a caching module that can have 3 functions, (or maybe a class containing methods). The primary 3 things we need are -
- download_from_cache()
- upload_to_cache()
- check_credentials()
There can be several ways to have this, but it should at least fulfill the following -
- Compare optimizer_version
- Compare file_version
Optional things can be to check file upload date and discard if it's older than a specified amount of time. If we go for a class based approach, we can also explore possibilities to improve performance.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing kiwix_storagelib usage in the scrapers and how optimizer and file versions are stored as metadata. Define the cache entry point and verify that credentials, downloads, uploads, and both version comparisons are covered; no specific files or tests are named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100