Synonyms with locale: "th" never match when the synonym term contains SARA AM (U+0E33) at a token boundary

Description

When a synonym term is written with the Thai character SARA AM (ำ, U+0E33) and the synonym item has locale: "th", the synonym never matches — not even for a single-token query typed exactly as the synonym was defined.

The same synonym works correctly when locale is omitted, and it also works if the synonym term is stored with SARA AM written in its decomposed form (ํ U+0E4D + า U+0E32).

This suggests the query token is normalized/decomposed under the th locale, but the synonym term is not normalized the same way when it is stored or looked up, so the two can never meet.

Environment

• Typesense v30.2, self-hosted (GET /debug → {"state":1,"version":"30.2"})
• Reproduced consistently across many runs on a fresh collection + fresh synonym set

Steps to reproduce

Thai is written as \uXXXX (body) and percent escapes (query string) so the script is
shell-encoding independent. The term used is น้ำหอม — "perfume".

TS=http://localhost:8108
KEY=xyz

# 1. Synonym set: "น้ำหอม" <-> "perfume", with locale: th
curl -s -X PUT "$TS/synonym_sets/bug_set" \
  -H "X-TYPESENSE-API-KEY: $KEY" -H 'Content-Type: application/json' \
  -d '{"items":[{"id":"s","synonyms":["\u0e19\u0e49\u0e33\u0e2b\u0e2d\u0e21","perfume"],"locale":"th"}]}'

# 2. Collection with a th-locale string field
curl -s -X POST "$TS/collections" \
  -H "X-TYPESENSE-API-KEY: $KEY" -H 'Content-Type: application/json' \
  -d '{"name":"bug","synonym_sets":["bug_set"],
       "fields":[{"name":"t","type":"string","locale":"th"}]}'

# 3. One document containing the English side of the synonym
curl -s -X POST "$TS/collections/bug/documents/import?action=create" \
  -H "X-TYPESENSE-API-KEY: $KEY" \
  -d '{"t":"item perfume"}'

# 4. Query with the Thai side of the synonym
curl -s "$TS/collections/bug/documents/search?q=%E0%B8%99%E0%B9%89%E0%B8%B3%E0%B8%AB%E0%B8%AD%E0%B8%A1&query_by=t&drop_tokens_threshold=0" \
  -H "X-TYPESENSE-API-KEY: $KEY"

Step 1 round-trips correctly — GET /synonym_sets/bug_set returns the term unchanged, so the
term is stored fine; only lookup fails.

Drop "locale":"th" from step 1 and step 2 and the same script returns found: 1.

Expected

found: 1 — the synonym น้ำหอม ↔ perfume should match the document item perfume.

Actual

found: 0

Isolating the cause

Same document, same synonym pair, only the encoding of SARA AM and locale change:

synonym term stored as	query typed as	locale	found
น้ำหอม composed (U+0E33)	น้ำหอม composed	(unset)	1 ✔
น้ำหอม composed (U+0E33)	น้ำหอม composed	th	0 ✘
น้ําหอม decomposed (U+0E4D U+0E32)	น้ำหอม composed	th	1 ✔
น้ำหอม composed (U+0E33)	น้ําหอม decomposed	th	0 ✘
น้ําหอม decomposed (U+0E4D U+0E32)	น้ําหอม decomposed	th	1 ✔

The query form does not matter. Only the stored synonym form does — a synonym term containing composed U+0E33 is unreachable under locale: "th".

Which words are affected

Not every word containing ำ breaks. With locale: "th", one synonym pair per row, single-token query:

Thai synonym term	found	note
น้ำหอม (perfume)	0 ✘	segments as น้ำ | หอม
น้ำตาล (sugar)	0 ✘	segments as น้ำ | ตาล
น้ำมัน (oil)	0 ✘	segments as น้ำ | มัน
ค่ำคืน (nighttime)	0 ✘	segments as ค่ำ | คืน
ซ้ำซาก (repetitive)	0 ✘	segments as ซ้ำ | ซาก
คำถาม (question)	0 ✘	segments as คำ | ถาม
ทำงาน (work)	0 ✘	segments as ทำ | งาน
น้ำ (water)	1 ✔	single token, ำ ends the string
ค่ำ (evening)	1 ✔	single token, ำ ends the string
กำไล (bangle)	1 ✔	single token, ำ is word-internal
น้ำแข็ง (ice)	1 ✔	kept as a single token by the segmenter
ชาแนล, ลิปสติก, รถยนต์	1 ✔	no ำ
เครื่องดื่ม, ผงซักฟอก, กระเป๋าเดินทาง	1 ✔	multi-token, but no token ends in ำ

Every failing case is one where the Thai segmenter puts a token boundary immediately after a SARA AM. Multi-token synonyms are fine as long as no token ends in ำ, and single tokens are fine even when they contain ำ.

Hypothesis: U+0E33 expands to two code points when decomposed, so a token boundary computed on the original string no longer lines up after normalization, corrupting the synonym term at exactly that boundary.

Real-world impact

Thai is written without spaces, so a large share of Thai search terms are compounds that the segmenter splits, and น้ำ... ("water-") is an extremely common prefix — น้ำหอม perfume, น้ำตาล sugar, น้ำมัน oil, น้ำยา cleaner, น้ำผลไม้ juice.

On our production catalog this silently reduced a two-word query from 182 matching products to 5: one token resolved through its synonym, the other did not, and the AND of the two collapsed the result set. Nothing in the response indicated a problem — no error, no warning, just a much smaller found.

The same catalog on Algolia returns 181 for the equivalent query, which is how the gap was noticed.

Workaround

Store the synonym term with SARA AM written out as ํ + า (U+0E4D U+0E32). Queries typed the normal way (U+0E33) then match. This is what our data already contained as a hand-added variant — ["น้ำหอม","น้ํา หอม","น้ําหอม"] — which is presumably someone hitting this bug and papering over it without knowing why.

Related

#1592 reports that a query containing two synonyms at once does not match. I could not reproduce that issue's original English case on 30.2 — {ave, avenue} + {sw, southwest} with q=ave sw correctly returns the Avenue Southwest documents. So this looks like a distinct, locale-specific failure rather than the same defect, but the two produce the same user-visible symptom and #1592 may well be the Thai/locale case in disguise.

Run the provided curl reproduction against a fresh Typesense collection and synonym set, then trace synonym lookup for a Thai-locale field and the token boundary after U+0E33. Done means composed SARA AM terms such as น้ำหอม match the English synonym under locale: "th", while the documented decomposed-form workaround remains unnecessary.