bitcoinsearch / bitcoinsearch/scraper

Delving Bitcoin Scraper Ignores Updates to Existing Posts

Open
#97 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
9
Forks
14
PR merge metrics
No merged PRs in 30d

Description

The current Delving Bitcoin scraper skips indexing when a document with the same ID already exists. This means updates to existing posts are not accounted for.

When adding Delving Bitcoin to `scraperv2`, we should address this issue and consider the following approaches:

#### Approach 1: Leverage Metadata Stored in ES Index
James' `discourse-archive` uses an on-disk `.metadata.json` file to store the `last_sync_date`, allowing it to avoid syncing from scratch each time. Currently, we are not using this functionality. Each GitHub Action workflow syncs everything from the beginning, which is manageable since Delving Bitcoin's history is relatively short.

In `scraperv2`, metadata from scrape jobs is stored in our Elasticsearch (ES) index. We can take advantage of this existing functionality to manage `last_sync_date` and only sync new or updated posts. However, this may require modifications to James' initial code to integrate this metadata mechanism.

#### Approach 2: Use a Custom Delving Bitcoin Archive
Another option is to forego using `discourse-archive` on every scraping run. Instead, we could maintain our own instance of the [delving-bitcoin-archive](https://github.com/jamesob/delving-bitcoin-archive) and read the necessary data directly from a GitHub repository. This would be similar to how we handle data in the [`GitHubMetadataScraper`](https://github.com/bitcoinsearch/scraper/blob/master/scraper/scrapers/github_metadata.py).

**Related**
- The current code in `delvingbitcoin_2_elasticsearch/achieve.py` is extracted from [jamesob/discourse-archive](https://github.com/jamesob/discourse-archive).
- https://delvingbitcoin.org/t/public-archive-for-delving-bitcoin/87/5
- https://meta.discourse.org/t/fetch-all-posts-from-a-topic-using-the-api/260886
- https://h-rd.org/archiving-a-discourse-forum/

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.