algolia / algolia/docsearch-scraper
Algolia index records reduction after an undefined amount of time
- Dominant language
- Python
- Stars
- 322
- Forks
- 115
- PR merge metrics
- No merged PRs in 30d
Description
### Situation
docsearch-scraper docker image starts scraping my company [documentation](https://docs.mia-platform.eu/) website any time we deploy a new version of it and correctly updates our indexes. Using the search bar will then return the correct records.
The problem is that, after an undefined amount of time (can be weeks or even months), our algolia search bar will not return any record if used. Is it possible that indexes get deleted after a certain amount of time or that algolia docsearch-scraper deletes them or returns no indexes if some condition occurs (maybe launching it multiple times in a small amount of time)?
### Result
Our documentation search through Algolia does not return any single record untill we run the docsearch-scraper docker image again.
### Workaround
Any time that we notice that our Algolia search does not return any result we run docsearch-scraper docker image again.
Contributor guide
Research direction
Start by reviewing the docsearch-scraper Docker image and the Algolia index lifecycle described in the report, then reproduce the loss after a deployment. Check whether scraper logs show index deletion or an empty update; done means identifying a reproducible condition and documenting a confirmed cause or fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100