sphinx-doc / sphinx-doc/sphinx

shortanchor compression is lossy

Open
#10,380 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

html search type:bug
Dominant language
Python
Stars
8k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

When the Sphinx search module encodes the search index, it uses a compression scheme for anchor names. Unfortunately this compression scheme:

https://github.com/sphinx-doc/sphinx/blob/c7e7cd0ebf4ed5ced75eb0511b34b837d487ab88/sphinx/search/__init__.py#L326

Unfortunately this compression scheme is lossy for a few reasons:

  1. The compression is based on comparing the HTML-escaped fullname to the anchor name. However, the fullname is never stored in the search index, only the dispname. The JavaScript search implementation therefore has to decode the anchor name based on the dispname, rather than the fullname, and if they differ then this would lead to the wrong result. For some domains, like Python, the dispname is always equal to the fullname, but for other domains, like std, that is not always the case.
  2. The compression uses the special values of "" and "-" to encode compressed values. However, that means that it cannot encode an actual anchor equal to "" or "-". For example, I have some Sphinx customizations that use a separate page for each object, and therefore have an empty string as the anchor name.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at sphinx/search/init.py around the linked compression code, then trace the JavaScript search decoding that consumes the index. Reproduce cases where dispname differs from fullname and where the anchor is empty or "-", and verify that the revised encoding and decoding preserve each anchor.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, python
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.