openzim / openzim/python-scraperlib

Add S3 based optimization cache support

Open
#30 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement stale
Dominant language
Python
Stars
31
Forks
27
Avg merge
3d 7h
Merged PRs (30d)
2

Description

We use kiwix_storagelib for implementing S3 based optimization cache in the scrapers. However, this gives rise to redundant code. We put a version of the file along with the optimizer version as the metadata always. So, this can be better implemented in scraperlib. For a start, we can have a caching module that can have 3 functions, (or maybe a class containing methods). The primary 3 things we need are -

  • download_from_cache()
  • upload_to_cache()
  • check_credentials()

There can be several ways to have this, but it should at least fulfill the following -

  • Compare optimizer_version
  • Compare file_version

Optional things can be to check file upload date and discard if it's older than a specified amount of time. If we go for a class based approach, we can also explore possibilities to improve performance.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing kiwix_storagelib usage in the scrapers and how optimizer and file versions are stored as metadata. Define the cache entry point and verify that credentials, downloads, uploads, and both version comparisons are covered; no specific files or tests are named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.