algolia / algolia/docsearch-scraper
Partial index promotion due to scrappy spider signals not being handled
- Dominant language
- Python
- Stars
- 322
- Forks
- 115
- PR merge metrics
- No merged PRs in 30d
Description
I have personally experienced `Ctrl-C` resulting in an incomplete index.
The scrappy documentation for `spider_closed` signal, https://docs.scrapy.org/en/latest/topics/signals.html#scrapy.signals.spider_closed , mentions that the reason for the closing should be `finished` under normal circumstances. However the reason could be `shutdown` in the case of a `Ctrl-C` signal stopping the spider.
It doesn't appear the that doc spider is patching into any of the spider signals. In the case that a spider does not finish naturally the scraper is unaware and will proceed to promote the incomplete temp index.
Contributor guide
Research direction
Inspect the spider signal handling and the temporary-index promotion path described in the issue. Reproduce a Ctrl-C shutdown and compare it with a normally finished spider. Done means a spider that closes for a non-finished reason cannot promote its incomplete temporary index.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100