Allow overriding the Site Search index analyzer via config to support CJK/RTL languages
@swicken is already working on this.
Since Jul 31, 2026.
- Dominant language
- Java
- Stars
- 970
- Forks
- 486
- Avg merge
- 3d 33m
- Merged PRs (30d)
- 170
Description
Description
Site Search hardcodes an English-tuned analyzer chain into its bundled index resources (es/os-sitesearch-settings.json + es/os-sitesearch-mapping.json): standard tokenizer + lowercase + asciifolding + a hardcoded English stemmer on the content field, with title/description/author falling back to the plain standard analyzer. There is no supported way to change tokenization short of repackaging the webapp.
This makes Site Search a poor fit for non-Latin languages:
- CJK text degrades to single-character tokens — a query for 東京 matches any document containing 東 or 京 anywhere, so precision and ranking are near-meaningless.
- RTL (Arabic/Hebrew) gets no normalization or stemming —
كتابdoes not matchالكتاب— and the English stemmer/asciifolding actively work against non-Latin text.
Elasticsearch/OpenSearch already ship ~35 built-in language analyzers (cjk, arabic, thai, ...) that are referenced by name with no plugin or settings declaration, and plugins add dictionary-grade ones (kuromoji, nori, smartcn). The gap is purely that Site Search offers no hook to select one.
Proposal: a single SITE_SEARCH_ANALYZER Config property (env: DOT_SITE_SEARCH_ANALYZER) naming any analyzer known to the cluster. When set, it is applied to the text fields (content, title, description, author) of the site-search mapping at index creation, for both the Elasticsearch and OpenSearch providers. Unset keeps the bundled defaults byte-for-byte. Deliberately not a settings/mapping file override — one env var covers every named analyzer without customers authoring engine JSON.
Acceptance Criteria
- Setting
DOT_SITE_SEARCH_ANALYZER(e.g.cjk,arabic) applies that analyzer to thecontent,title,description, andauthorfields of newly created Site Search indexes on both the ES and OS providers - The
content.ngramsubfield'ssearch_analyzerfollows the override so query-time tokenization stays consistent - With the property unset, index creation behavior and the bundled settings/mapping are unchanged
- An analyzer name unknown to the cluster fails index creation loudly with the engine's error, rather than silently building an index with default analyzers
- The override takes effect at index creation (existing indexes require a rebuild); documented in the code
- Unit tests cover the default (untouched) path, the override applied to text fields only, and missing-resource failure
Priority
Medium
Additional Context
- Requested by a partner needing best-practice guidance for RTL and CJK site search.
- Built-in analyzers (
cjk,arabic,thai, ...) require nothing on the cluster; plugin analyzers (kuromoji,nori,smartcn) additionally require the plugin installed on every node of the ES/OS cluster. - Known limitation (accepted): one analyzer per index — mixed-language sites pick a single tokenization. Per-language subfields/indexes are out of scope until a customer needs them.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.