dotnet / dotnet/docfx

[Bug] Search Tokenizer Config not correctly applied from lunr.tokenizer.separator config

Open
#9,682 0 comments 0 reactions 0 assignees View on GitHub
search
Dominant language
C#
Stars
4.4k
Forks
890
Avg merge
2h 11m
Merged PRs (30d)
10

Description

According to the fix (https://github.com/dotnet/docfx/pull/5083) and the code in default template and modern template the lunr.tokenizer.separator should be whitespace, dash, dot and round brackets. The config for default template looks ok, the config for modern template is not escaped: lunr.tokenizer.separator = /[\s\-.()]+/

default template:
https://github.com/dotnet/docfx/blob/bd56627e35cd5d0194e115c3e09835b93e84f9a1/templates/default/src/search-worker.js#L9

modern template:
https://github.com/dotnet/docfx/blob/bd56627e35cd5d0194e115c3e09835b93e84f9a1/templates/modern/src/search-worker.ts#L46

Furthermore, when you build the project there will :
_site/public/search-worker.min.js which contains `t.tokenizer.separator=/[\s\-]+/` and `w.default.tokenizer.separator=/[\s\-.()]+/`

i think both are not correct and the regex should be in both cases: `/[\s\-\.\(\)]+/`

But when you modify the _site/public/search-worker.min.js and fix the first usage to `t.tokenizer.separator=/[\s\-\.\(\)]+/` search terms with dots will now work.

**To Reproduce**
Steps to reproduce the behavior:
1. run docfx init
2. _enableSearch should be set to true
3. open docs/getting-started.md and add some text like: filename.jpg
4. run docfx build and serve the page
6. if you search for "filename" it will find your text, but not if you type "filename.jpg".
7. examine the generated search-worker.min.js file and check the regex pattern.

![docfx-separator-withdot](https://github.com/dotnet/docfx/assets/8428000/daffc733-dfac-4afc-90c2-2bebfa7b7da1)
![docfx-separator](https://github.com/dotnet/docfx/assets/8428000/b5685c81-0352-486b-bffa-94993a376546)

**Expected behavior**
When entering filename.jpg in the search form field it should tokenize at the dot.
The generated _site/public/search-worker.min.js file should define the pattern as `t.tokenizer.separator=/[\s\-\.\(\)]+/` and fixing the default by escaping the characters in the regex: `w.default.tokenizer.separator=/[\s\-\.\(\)]+/`
After that change and clearing the browser cache a search for "filname.jpg" matches the text.

**Context (please complete the following information):**
- OS: Windows
- Docfx version: 2.75.2+fe673ecea2ac444a4fd480e6cfcf605c78614385

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.