openzim / openzim/python-scraperlib

Enhancement: Inversion of control scraper base class

Aperta
#195 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

question
Lingua principale
Python
Stelle
31
Fork
27
Merge medio
3g 7h
PR unite (30g)
2

Descrizione

When recently building the DevDocs scraper, I realized there are a ton of things that I was relying on @benoit74's expertise for to make the scraper sustainable for ZimFarm but weren't core to building a functional scraper. These include:

  • Caching if S3 is available.
  • Expected CLI flags.
  • Zim file parameters/naming/destinations.
  • ZimFarm operator friendly logging setup.
  • Progress tracker.
  • HTTP client setup.

The scraper still isn't fully there, but I've already spent a lot of time implementing and testing some of these things. I've written Logstash plugins and Elastic Beats (which are more smiilar to Zim scrapers and those were dramatically easier because that common logic was abstracted away.

I'd love to see something like the following (example only!) as what I had to build while knowing most of the above would be taken care of:

# New type ZimMetadata contains the properties to populate config_metadata() on a Zim.
# There are specific types based on whether the scraper is for a single Zim or multiple (in which case
# it supports placeholders).
# Methods can be overridden for fine-grained control e.g. to add additional formatting parameters.
M = TypeVar('M', bound=ZimMetadata) 

class MyScraper(MultiZimScraper):

  # Parent class includes a logger, HTTP client, potentially other items.

  def add_flags(parser: argparse.ArgumentParser):
    '''Add custom flags to the program'''
    pass
  
  def setup(namespace: argparse.Namespace):
    '''Parse flags and set up resources for execution.
    After this call, MultiZimScraper may have additional internal variables set up
    e.g. an HTTP client that automatically caches to S3 if running in ZimFarm and with
    retries/delay.
    '''
    pass
  
  def list_zims() -> M:
    '''Called after setup to list all Zims to be created. '''
    pass
  
  def add_contents(creator: Creator, metadata: M):
    '''Called for each item in list_zims().
    
    The JSON progress file is updated between calls, logs for progress/next ZIM/timing are written
    and a scraper check utility could be asserted after.
    '''
    pass

I don't think all scrapers would need to use this format, but something like it would have dramatically cut down on the amount of testing and knowledge needed for me to produce a quality Zim scraper.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Direzione di ricerca

Inizia esaminando le astrazioni degli scraper esistenti e il lavoro sullo scraper DevDocs descritto nell’issue; non vengono indicati file o test specifici. Definisci una classe base riutilizzabile che copra la configurazione condivisa, come il caching, i flag CLI, i metadati Zim, il logging, il monitoraggio dell’avanzamento e l’accesso HTTP, quindi verifica che uno scraper possa utilizzarla con meno configurazione e test duplicati.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
tooling
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.