dotCMS / dotCMS/core

Search queries are lowercased end to end, so case-sensitive matching is impossible and quoted phrases are split

Open
#37,522 0 comments 0 reactions 1 assignee View on GitHub

@fabrizzio-dotCMS is already working on this.

Since Sep 11, 2026.

Team : Scout Type : Task
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Description

Every query that goes through the search view tools and the search API is folded to lower case
before it runs, values included, so a case-sensitive match is impossible through them. Quoted
phrases are not protected either.

The folding is done by StringUtils.lowercaseStringExceptMatchingTokens(query, LUCENE_RESERVED_KEYWORDS_REGEX):

  • StringUtils.java:335 — the helper splits the query on spaces with a StringTokenizer and
    lower-cases every token that does not match the regex.
  • ESContentFactoryImpl.java:252 — the regex is "OR|AND|NOT|TO", so only those four tokens survive.

Call sites (both engines, so the behaviour is the same in every migration phase):

Path Where
Legacy ES search enterprise/.../priv/ESSearchAPIImpl.java:64 (esSearch) and :116 (esSearchRaw)
OpenSearch OSSearchAPIImpl.java:92 (search) and :154 (searchRaw)
What it does to a query
Written Executed
+title:Hello +title:hello
+title:"Hello World" split on the space, both halves folded — the quotes protect nothing
+contentType:News AND +live:true +contenttype:news AND +live:true (AND survives)

Two separate consequences:

  1. No case-sensitive matching. Against a keyword-mapped field holding News, a query for
    News is executed as news and does not match. This hits identifiers, product codes, tags with
    their own capitalisation, and any _dotraw field an exact match was expected on.
  2. Quoted phrases are tokenised by the helper. Splitting on spaces means a phrase query is
    processed as separate tokens before it ever reaches the engine, which is not what the author
    wrote.

The field-name half of this is deliberate and useful — it is what lets contentType resolve to the
physical index field contenttype. The value half looks like collateral damage rather than intent.

Not a migration regression

This dates to c4e2d072cc (2023-04-19) on the Elasticsearch side. The OpenSearch implementation
added the same call deliberately for parity (80b6a1bcc6, #35609) — its own comments say
"Reuses the existing lowercasing helper for parity" and "Symmetric with the ES raw path". So
nothing about it changes when an installation migrates; it is raised here because the ES→OS review
surfaced it, and because customer-facing documentation currently has to warn about it as a
permanent limitation.

Options to weigh
  1. Fold field names only, leaving the value after the first : untouched. Closest to the
    original intent, and the only option that actually enables case-sensitive matching.
  2. Protect quoted spans, so "Hello World" is passed through as written. Fixes consequence 2
    without touching consequence 1.
  3. Opt out per call — a flag on the view tool / API method, defaulting to today's behaviour.
    Safest for existing templates, but pushes the decision onto every caller.
  4. Document it and close. It is long-standing behaviour and anything else risks changing results
    for templates that have been written against it for two years.

Option 1 is the one worth costing first, but it cannot ship without an answer to: how many existing
templates rely on a mixed-case value being folded for them? That is the real risk, and it is
measurable — the audit endpoint proposed in #37520 could report it.

Acceptance Criteria

  • A decision is recorded on which option above we take, with the compatibility risk stated.
  • If behaviour changes: field names keep resolving case-insensitively (contentType
    contenttype) — that part must not regress.
  • If behaviour changes: a query for a mixed-case value against a keyword field matches, and a
    quoted phrase reaches the engine as written.
  • Tests cover both engines, since the two paths fold independently and could drift.
  • Whatever we decide, SEARCH_API_MIGRATION.md and the customer-facing migration guide state the
    final behaviour explicitly — today they have to describe it as a limitation.

Additional Context

Surfaced while verifying the Velocity examples in the customer ES→OpenSearch migration guide against
the code. Related: #37520 (VTL audit/rewrite endpoint), which would give us the usage numbers needed
to judge the compatibility risk.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.