sphinx-doc / sphinx-doc/sphinx
Word characters for search index should NOT include underscore
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8k
- Forks
- 2.6k
- PR merge metrics
- No merged PRs in 30d
Description
Describe the bug
When the search index is built, the text is split into words. This is done with the regex \w+ - line 85. \w includes characters, but also includes underscores and should not.
The consequence is that if your text has word1_word2_word3 then doing a search for word2 or word3 will not find that match. Underscore seperated words are common in Python and elsewhere, and the Javascript tokenizer does consider underscore as a splitter.
I experienced this with SQLITE_CONFIG_URI being in my doc, but searches for uri do not find it.
How to Reproduce
You can see this with Sphinx's own doc. Search for apply_source_workaround and you'll see apply_source_workaround found. Now search for workaround and apply_source_workaround is not found at all.
Environment Information
Platform: linux; (Linux-5.19.0-31-generic-x86_64-with-glibc2.36)
Python version: 3.10.7 (main, Nov 24 2022, 19:45:47) [GCC 12.2.0])
Python implementation: CPython
Sphinx version: 6.1.3
Docutils version: 0.19
Jinja2 version: 3.1.2
Pygments version: 2.14.0
Sphinx extensions
No response
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at sphinx/search/init.py line 85, where the search index splits text with the reported regex, and compare its behavior with the JavaScript tokenizer. Reproduce searches for apply_source_workaround and workaround, then verify that underscore-separated words can be searched individually without breaking the existing full-term search.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- search
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100