algolia / algolia/docsearch-scraper

Partial index promotion due to scrappy spider signals not being handled

Open
#540 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
322
Forks
115
PR merge metrics
No merged PRs in 30d

Description

I have personally experienced `Ctrl-C` resulting in an incomplete index.

The scrappy documentation for `spider_closed` signal, https://docs.scrapy.org/en/latest/topics/signals.html#scrapy.signals.spider_closed , mentions that the reason for the closing should be `finished` under normal circumstances. However the reason could be `shutdown` in the case of a `Ctrl-C` signal stopping the spider.

It doesn't appear the that doc spider is patching into any of the spider signals. In the case that a spider does not finish naturally the scraper is unaware and will proceed to promote the incomplete temp index.

Contributor guide

Open the contributing guide

Research direction

Inspect the spider signal handling and the temporary-index promotion path described in the issue. Reproduce a Ctrl-C shutdown and compare it with a normally finished spider. Done means a spider that closes for a non-finished reason cannot promote its incomplete temporary index.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.