algolia / algolia/docsearch-scraper

Ignore sidebar headings in algolia indexing

Open
#572 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
322
Forks
115
PR merge metrics
No merged PRs in 30d

Description

I am having a problem with algolia. I want to exclude sidebar headings from getting listed.

When I search `gallery` I am getting 6 results with no difference. Because I have a sidebar named `Website Gallery` and it has 6 items inside it. That's why the search is being done for all 6.

![gallery](https://user-images.githubusercontent.com/77567538/210197848-e1c95b0d-eb79-414d-8253-17622f67f481.png)

**Can you please tell me how can I ignore sidebar headings in `config.json`?**

My config.json is
```json
{
"index_name": "portfolio-generator",
"start_urls": ["https://portfolio-generator.cybersaksham.co.in/"],
"sitemap_urls": [
"https://portfolio-generator.cybersaksham.co.in/sitemap.xml"
],
"sitemap_alternate_links": true,
"stop_urls": ["/tests"],
"js_render": true,
"js_wait": true,
"selectors": {
"lvl0": {
"selector": "(//ul[contains(@class,'menu__list')]//a[contains(@class, 'menu__link menu__link--sublist menu__link--active')]/text() | //nav[contains(@class, 'navbar')]//a[contains(@class, 'navbar__link--active')]/text())[last()]",
"type": "xpath",
"global": true,
"default_value": "Documentation"
},
"lvl1": "header h1",
"lvl2": "article h2",
"lvl3": "article h3",
"lvl4": "article h4",
"lvl5": "article h5, article td:first-child",
"lvl6": "article h6",
"text": "article p, article li, article td:last-child"
},
"strip_chars": " .,;:#",
"custom_settings": {
"separatorsToIndex": "_",
"attributesForFaceting": ["language", "version", "type", "docusaurus_tag"],
"attributesToRetrieve": [
"hierarchy",
"content",
"anchor",
"url",
"url_without_anchor",
"type"
]
},
"conversation_id": ["833762294"],
"nb_hits": 9000
}
```

_Originally posted by @cybersaksham in https://github.com/algolia/docsearch-scraper/issues/571#issuecomment-1368670652_

Contributor guide

Open the contributing guide

Research direction

The issue names config.json and its selectors block; start by reading the selector configuration and how heading levels are collected. Verify the generated Algolia records for the supplied site, and consider the work complete when sidebar headings no longer create duplicate search results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
search
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.