sphinx-doc / sphinx-doc/sphinx

Word characters for search index should NOT include underscore

Open
#11,253 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

html search
Dominant language
Python
Stars
8k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

Describe the bug

When the search index is built, the text is split into words. This is done with the regex \w+ - line 85. \w includes characters, but also includes underscores and should not.

The consequence is that if your text has word1_word2_word3 then doing a search for word2 or word3 will not find that match. Underscore seperated words are common in Python and elsewhere, and the Javascript tokenizer does consider underscore as a splitter.

I experienced this with SQLITE_CONFIG_URI being in my doc, but searches for uri do not find it.

How to Reproduce

You can see this with Sphinx's own doc. Search for apply_source_workaround and you'll see apply_source_workaround found. Now search for workaround and apply_source_workaround is not found at all.

Environment Information
Platform:              linux; (Linux-5.19.0-31-generic-x86_64-with-glibc2.36)
Python version:        3.10.7 (main, Nov 24 2022, 19:45:47) [GCC 12.2.0])
Python implementation: CPython
Sphinx version:        6.1.3
Docutils version:      0.19
Jinja2 version:        3.1.2
Pygments version:      2.14.0
Sphinx extensions

No response

Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at sphinx/search/init.py line 85, where the search index splits text with the reported regex, and compare its behavior with the JavaScript tokenizer. Reproduce searches for apply_source_workaround and workaround, then verify that underscore-separated words can be searched individually without breaking the existing full-term search.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
search
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.