algolia / algolia/docsearch

Encoded special characters should be decoded by the crawler

Open
#1,723 0 comments 1 reaction 0 assignees View on GitHub
crawler
Dominant language
TypeScript
Stars
4.4k
Forks
439
Avg merge
14h 32m
Merged PRs (30d)
19

Description

## Description

When websites contain encoded special characters (e.g. `&` instead of `&`), they aren’t displayed correctly in the DocSearch UI.

## Steps to reproduce

1. Set up a website that uses encoded special characters (e.g. `&`) in a heading.
2. Use the `docsearch` helper in the record extractor to extract contents.
3. The encoded special characters are indexed.
4. When rendering the DocSearch UI, the record contents are encoded again which means that `&` is displayed rather than `&`.

You can test this on our documentation site: https://docs.aleph.occrp.org. The sidebar has a "Developers & Admins" section.

This is what it is rendered like in the DocSearch UI:
Screen Shot 2022-12-20 at 15 56 13

## Expected behavior

Algolia Crawler or the `docsearch` helper should take care of decoding encoded characters, e.g. by using the respective Cheerio configuration when loading the HTML or by decoding after extracting contents from the HTML.

I’ve adjusted the Crawler configuration and manually decode `&` for now, but it would be nice if either Aloglia Crawler or the `docsearch` helper could do this automatically.

```js
recordExtractor: ({ helpers }) => {
const data = helpers.docsearch({ /* ... */ });

data.forEach((record) => {
Object.entries(record.hierarchy)
.filter(([level, content]) => content)
.forEach(([level, content]) => {
record.hierarchy[level] = content.replace("&", "&");
});
});

return data;
}
```

## Environment

- OS: Any OS
- Browser: Any browser
- DocSearch version: 3.3.0

Contributor guide

Open the contributing guide

Research direction

Start by tracing the recordExtractor's helpers.docsearch path and the crawler's HTML parsing, focusing on where heading contents are extracted and encoded. Reproduce the Developers & Admins example and confirm that encoded characters are decoded once before indexing and display correctly in the DocSearch UI.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, typescript
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.