algolia / algolia/docsearch-scraper
Crawler does not seem to work on websites that use shadowDOM
Open
- Dominant language
- Python
- Stars
- 322
- Forks
- 115
- PR merge metrics
- No merged PRs in 30d
Description
Hello Algolia Devs,
I tried to add search function to [my website](https://gem-book.js.org), but I got the reply that no content can be crawled. Is it because my website uses shadowdom?
Thank you.
Contributor guide
Research direction
Start by reproducing the crawler failure against the linked website and checking whether its shadow DOM content is discovered. Determine whether shadow DOM is supported and document the required behavior or the limitation; the issue is done when the crawler’s behavior is confirmed and the result is recorded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, python
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100