algolia / algolia/docsearch-scraper

Crawler does not seem to work on websites that use shadowDOM

Open
#552 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
322
Forks
115
PR merge metrics
No merged PRs in 30d

Description

Hello Algolia Devs,

I tried to add search function to [my website](https://gem-book.js.org), but I got the reply that no content can be crawled. Is it because my website uses shadowdom?

Thank you.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the crawler failure against the linked website and checking whether its shadow DOM content is discovered. Determine whether shadow DOM is supported and document the required behavior or the limitation; the issue is done when the crawler’s behavior is confirmed and the result is recorded.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, python
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.