algolia / algolia/docsearch-scraper
Ignore sidebar headings in algolia indexing
- Dominant language
- Python
- Stars
- 322
- Forks
- 115
- PR merge metrics
- No merged PRs in 30d
Description
I am having a problem with algolia. I want to exclude sidebar headings from getting listed.
When I search `gallery` I am getting 6 results with no difference. Because I have a sidebar named `Website Gallery` and it has 6 items inside it. That's why the search is being done for all 6.

**Can you please tell me how can I ignore sidebar headings in `config.json`?**
My config.json is
```json
{
"index_name": "portfolio-generator",
"start_urls": ["https://portfolio-generator.cybersaksham.co.in/"],
"sitemap_urls": [
"https://portfolio-generator.cybersaksham.co.in/sitemap.xml"
],
"sitemap_alternate_links": true,
"stop_urls": ["/tests"],
"js_render": true,
"js_wait": true,
"selectors": {
"lvl0": {
"selector": "(//ul[contains(@class,'menu__list')]//a[contains(@class, 'menu__link menu__link--sublist menu__link--active')]/text() | //nav[contains(@class, 'navbar')]//a[contains(@class, 'navbar__link--active')]/text())[last()]",
"type": "xpath",
"global": true,
"default_value": "Documentation"
},
"lvl1": "header h1",
"lvl2": "article h2",
"lvl3": "article h3",
"lvl4": "article h4",
"lvl5": "article h5, article td:first-child",
"lvl6": "article h6",
"text": "article p, article li, article td:last-child"
},
"strip_chars": " .,;:#",
"custom_settings": {
"separatorsToIndex": "_",
"attributesForFaceting": ["language", "version", "type", "docusaurus_tag"],
"attributesToRetrieve": [
"hierarchy",
"content",
"anchor",
"url",
"url_without_anchor",
"type"
]
},
"conversation_id": ["833762294"],
"nb_hits": 9000
}
```
_Originally posted by @cybersaksham in https://github.com/algolia/docsearch-scraper/issues/571#issuecomment-1368670652_
Contributor guide
Research direction
The issue names config.json and its selectors block; start by reading the selector configuration and how heading levels are collected. Verify the generated Algolia records for the supplied site, and consider the work complete when sidebar headings no longer create duplicate search results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- search
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100