sphinx-doc / sphinx-doc/sphinx

`json` builder incorrectly caches translation when building multiple languages

Open
#13,448 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

builder:json type:bug
Dominant language
Python
Stars
8k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

Describe the bug

I have a project that uses Sphinx as a library (using sphinx.application.Sphinx) and that builds documentation in several different languages.
We noticed that some translated strings would incorrectly persist between multiple builds. That is to say, building language A then language B would result in documentation B having some strings from language A.
Adding a combination of patch_docutils, docutils_namespace, and _clean_up_global_state helped, but not completely. In particular the string "Python Module Index" would keep whichever translation it had in the first built language.

I believe the cause of this issue is two-fold:

  1. The custom json encoder in sphinxcontrib.serializinghtml was not updated correctly after the refactor in 363cdc090633b41e9efc663087f0b862947daf6d (lazily translated strings are no longer a subclass of UserString)
  2. The string "Python Module Index" comes from a class-level variable (sphinx.domains.python.PythonModuleIndex.localname) and is therefore cached the first time that module is imported. If a language is active at the time of import, the translation will stay cached even if a different language is activated.
How to Reproduce

Our translation infrastructure is fairly complex, but I managed to narrow down the bug to this testcase:

Click to unfold (80 lines)
from shutil import rmtree
from pathlib import Path
import json
import sys

from sphinx.application import Sphinx

from sphinx.locale import _TranslationProxy
from sphinx.testing.util import _clean_up_global_state
from sphinx.util.docutils import docutils_namespace, patch_docutils
from sphinxcontrib.serializinghtml import jsonimpl

CURRENTDIR = Path(__file__).parent
SOURCEDIR = CURRENTDIR / "docs"
BUILDDIR = CURRENTDIR / "build"


class FixedSphinxJSONEncoder(jsonimpl.SphinxJSONEncoder):
    def default(self, obj):
        # Handle _TranslationProxy correctly
        if isinstance(obj, _TranslationProxy):
            return str(obj)
        return super().default(obj)


def build(lang):
    with patch_docutils(SOURCEDIR), docutils_namespace():
        Sphinx(
            srcdir=SOURCEDIR,
            outdir=BUILDDIR / lang,
            doctreedir=BUILDDIR / lang / ".doctrees",
            confdir=None,
            warning=None,
            status=None,
            buildername="json",
            confoverrides={"language": lang},
        ).build()
    # Clean up global state after building each language.
    _clean_up_global_state()


def test():
    with (BUILDDIR / "en" / "py-modindex.fjson").open() as f:
        data = json.load(f)

    if (s := data["indextitle"]) == "Python Module Index":
        print("Looks like the bug was fixed, yay!")
        sys.exit(0)
    else:
        print(f"Incorrect translation detected: {s}")
        sys.exit(1)


def apply_fix():
    # An early-loading of the module forces `PythonModuleIndex.localname` to
    # be a _TranslationProxy object and not a string
    from sphinx.domains import python

    # Monkeypatch the custom JSON encoder to handle translation proxies correctly
    jsonimpl.SphinxJSONEncoder = FixedSphinxJSONEncoder


if __name__ == "__main__":
    # Clean up source and build dir (from previous runs)
    rmtree(BUILDDIR, ignore_errors=True)
    rmtree(SOURCEDIR, ignore_errors=True)
    SOURCEDIR.mkdir()

    # Create a single .rst file containing a :py:module directive
    (SOURCEDIR / "index.rst").write_text(".. py:module:: sphinxwtf")

    # Uncomment to apply the workaround fix
    # apply_fix()

    for lang in ["fr", "en"]:
        print(f"Building {lang}")
        build(lang)

    test()

This script should be run somewhere where sphinx is installed, or if you have uv you can use this oneliner (assuming you saved the code as script.py):

uv run --with sphinx script.py

(Note that this script will generate two directories docs and build in the current directory)

You can see that the script builds the same document first in French, then in English. Then it loads a json file from the English build and checks whether the "Python Module Index" string is correctly translated. I've included an apply_fix() function that fixes the issue in two steps.

Environment Information
Platform:              linux; (Linux-6.13.7-arch1-1-x86_64-with-glibc2.41)
Python version:        3.13.2 (main, Feb  5 2025, 08:05:21) [GCC 14.2.1 20250128])
Python implementation: CPython
Sphinx version:        8.2.3
Docutils version:      0.21.2
Jinja2 version:        3.1.6
Pygments version:      2.19.1
Sphinx extensions
No extensions are involved in this bug.
Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the supplied two-language reproduction, then inspect sphinxcontrib.serializinghtml.jsonimpl.SphinxJSONEncoder and sphinx.domains.python.PythonModuleIndex.localname, using the _TranslationProxy handling shown in apply_fix(). Done means building French followed by English produces the correct English value for indextitle in py-modindex.fjson without the workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, localization
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.