bitcoinsearch / bitcoinsearch/scraper

Schedule Regular Scraping of Bitcoin Improvement Proposals (BIPs, BLIPs, BOLTs)

Open
#79 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
9
Forks
14
PR merge metrics
No merged PRs in 30d

Description

Currently, we have scraping code for Bitcoin Improvement Proposals ([BIPs](https://github.com/bitcoinsearch/scraper/blob/master/scrapybot/scrapybot/spiders/bips.py), [BLIPs](https://github.com/bitcoinsearch/scraper/blob/master/scrapybot/scrapybot/spiders/blips.py), and [BOLTs](https://github.com/bitcoinsearch/scraper/blob/master/scrapybot/scrapybot/spiders/bolts.py)) from GitHub repositories, but it isn't set to run on a schedule. As a result, these sources are not being updated regularly and may contain outdated information.

We need to implement a cron job to ensure that these improvement proposals are scraped on a more frequent basis, so that our data stays current.

Tasks:
1. Create a cron job for regular scraping of BIPs, BLIPs, and BOLTs.
2. Define an appropriate frequency for scraping (e.g., daily or weekly).
3. Ensure the scraped data is stored properly in the Elasticsearch index.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.