dotCMS / dotCMS/core

Allow overriding the Site Search index analyzer via config to support CJK/RTL languages

Open
#36,839 4 comments 0 reactions 1 assignee View on GitHub

@swicken is already working on this.

Since Jul 31, 2026.

Doc : Needs Doc dotCMS : Localization Team : Platform Type : Task
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Description

Site Search hardcodes an English-tuned analyzer chain into its bundled index resources (es/os-sitesearch-settings.json + es/os-sitesearch-mapping.json): standard tokenizer + lowercase + asciifolding + a hardcoded English stemmer on the content field, with title/description/author falling back to the plain standard analyzer. There is no supported way to change tokenization short of repackaging the webapp.

This makes Site Search a poor fit for non-Latin languages:

  • CJK text degrades to single-character tokens — a query for 東京 matches any document containing 東 or 京 anywhere, so precision and ranking are near-meaningless.
  • RTL (Arabic/Hebrew) gets no normalization or stemming — كتاب does not match الكتاب — and the English stemmer/asciifolding actively work against non-Latin text.

Elasticsearch/OpenSearch already ship ~35 built-in language analyzers (cjk, arabic, thai, ...) that are referenced by name with no plugin or settings declaration, and plugins add dictionary-grade ones (kuromoji, nori, smartcn). The gap is purely that Site Search offers no hook to select one.

Proposal: a single SITE_SEARCH_ANALYZER Config property (env: DOT_SITE_SEARCH_ANALYZER) naming any analyzer known to the cluster. When set, it is applied to the text fields (content, title, description, author) of the site-search mapping at index creation, for both the Elasticsearch and OpenSearch providers. Unset keeps the bundled defaults byte-for-byte. Deliberately not a settings/mapping file override — one env var covers every named analyzer without customers authoring engine JSON.

Acceptance Criteria

  • Setting DOT_SITE_SEARCH_ANALYZER (e.g. cjk, arabic) applies that analyzer to the content, title, description, and author fields of newly created Site Search indexes on both the ES and OS providers
  • The content.ngram subfield's search_analyzer follows the override so query-time tokenization stays consistent
  • With the property unset, index creation behavior and the bundled settings/mapping are unchanged
  • An analyzer name unknown to the cluster fails index creation loudly with the engine's error, rather than silently building an index with default analyzers
  • The override takes effect at index creation (existing indexes require a rebuild); documented in the code
  • Unit tests cover the default (untouched) path, the override applied to text fields only, and missing-resource failure

Priority

Medium

Additional Context

  • Requested by a partner needing best-practice guidance for RTL and CJK site search.
  • Built-in analyzers (cjk, arabic, thai, ...) require nothing on the cluster; plugin analyzers (kuromoji, nori, smartcn) additionally require the plugin installed on every node of the ES/OS cluster.
  • Known limitation (accepted): one analyzer per index — mixed-language sites pick a single tokenization. Per-language subfields/indexes are out of scope until a customer needs them.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.