openzim / openzim/python-scraperlib
Enhancement: Inversion of control scraper base class
还没有人认领这个 Issue。
- 主要语言
- Python
- 星标
- 31
- 派生
- 27
- 平均合并
- 3 天 7 小时
- 30 天内合并 PR
- 2
描述
When recently building the DevDocs scraper, I realized there are a ton of things that I was relying on @benoit74's expertise for to make the scraper sustainable for ZimFarm but weren't core to building a functional scraper. These include:
- Caching if S3 is available.
- Expected CLI flags.
- Zim file parameters/naming/destinations.
- ZimFarm operator friendly logging setup.
- Progress tracker.
- HTTP client setup.
The scraper still isn't fully there, but I've already spent a lot of time implementing and testing some of these things. I've written Logstash plugins and Elastic Beats (which are more smiilar to Zim scrapers and those were dramatically easier because that common logic was abstracted away.
I'd love to see something like the following (example only!) as what I had to build while knowing most of the above would be taken care of:
# New type ZimMetadata contains the properties to populate config_metadata() on a Zim.
# There are specific types based on whether the scraper is for a single Zim or multiple (in which case
# it supports placeholders).
# Methods can be overridden for fine-grained control e.g. to add additional formatting parameters.
M = TypeVar('M', bound=ZimMetadata)
class MyScraper(MultiZimScraper):
# Parent class includes a logger, HTTP client, potentially other items.
def add_flags(parser: argparse.ArgumentParser):
'''Add custom flags to the program'''
pass
def setup(namespace: argparse.Namespace):
'''Parse flags and set up resources for execution.
After this call, MultiZimScraper may have additional internal variables set up
e.g. an HTTP client that automatically caches to S3 if running in ZimFarm and with
retries/delay.
'''
pass
def list_zims() -> M:
'''Called after setup to list all Zims to be created. '''
pass
def add_contents(creator: Creator, metadata: M):
'''Called for each item in list_zims().
The JSON progress file is updated between calls, logs for progress/next ZIM/timing are written
and a scraper check utility could be asserted after.
'''
pass
I don't think all scrapers would need to use this format, but something like it would have dramatically cut down on the amount of testing and knowledge needed for me to produce a quality Zim scraper.
贡献指南
这个仓库没有索引到贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
首先查看现有的 scraper 抽象以及 issue 中描述的 DevDocs scraper 工作;未指定具体文件或测试。定义一个可复用的基类,涵盖缓存、CLI 标志、Zim 元数据、日志记录、进度跟踪和 HTTP 访问等共用设置,然后验证 scraper 是否可以使用它,从而减少重复的设置和测试。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- tooling
- Issue 类型
- 功能
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 25/100