uttrflow / uttrflow/uttrflow-swift
The suggestion store never normalises Unicode, so the same Hindi or accented line typed through two input sources is stored twice and one spelling can't find the other
- Dominant language
- Swift
- Stars
- 4
- Forks
- 17
- Avg merge
- 3h 32m
- Merged PRs (30d)
- 277
Description
## What happens
No code under `Sources/` normalises text: there is no `precomposedStringWithCanonicalMapping`, `decomposedStringWithCanonicalMapping` or equivalent. Swift's `String ==` compares canonically equivalent strings as equal, but SQLite compares bytes. The suggestion store works in SQL:
- the prefix scan: `text_lower >= ? AND text_lower < ?` (`Sources/UttrflowPredictStore/PredictStore.swift:165-186`);
- uniqueness: `UNIQUE (surface_id, text)` (`Sources/UttrflowPredictStore/Schema.swift:38`);
- the fuzzy fallback, over UTF-8 bytes (`PredictStore.swift:189-215`).
Devanagari nukta letters have two encodings: ज़ is U+095B, or ज U+091C + ़ U+093C. Latin accents do too: é is U+00E9, or e + U+0301. Which one arrives depends on the keyboard layout, the input method and where text was pasted from. Checked with the `sqlite3` tool:
```
create table e(t text, unique(t));
insert into e values(char(0x95B)||'िंदगी'); -- ज़िंदगी, precomposed
insert into e values(char(0x91C,0x93C)||'िंदगी'); -- ज़िंदगी, decomposed: accepted, 2 rows
select count(*) from e where t >= char(0x91C,0x93C) and t < char(0x91C,0x93D); -- 1
```
In Swift the two strings are `==`, but their UTF-8 differs. So:
- the same line is learned twice, and its count and recency are split between the two rows;
- a prefix typed in one form doesn't find the line stored in the other.
`Surroundings.cleaned` (`Sources/UttrflowContext/Surroundings.swift:197-203`) strips every `.format` scalar from screen context. That includes ZWJ and ZWNJ (U+200D, U+200C), which Hindi typing uses to pick a half-form. Stored lines keep them. Only the prompt context is affected, which is minor.
## Why it matters
Hindi users commonly switch between a phonetic input method and a keyboard layout, and paste from messaging apps. Each can produce a different encoding of the same word, most visibly with nukta letters (ज़, फ़, क़, ड़ are everyday letters). Their suggestions then learn more slowly and miss matches an English user would get.
## Acceptance criteria
- Text is normalised to NFC in one place, before it is stored and before it is matched: the store's `learn`/record paths and the typed prefix in `exactCandidates`/`fuzzyCandidates`.
- Existing rows are normalised once by a schema migration. Rows that become duplicates are merged, adding their counts.
- ZWJ and ZWNJ are kept when context is cleaned (strip other `.format` scalars as today).
- Tests in `Tests/UttrflowPredictStoreTests/PredictStoreTests.swift` (or `CaseMatchingTests.swift`): the precomposed and decomposed forms of "ज़िंदगी" and "café" are one entry and find each other.
Contributor guide
Research direction
Start with the SQL paths in Sources/UttrflowPredictStore/PredictStore.swift:165-215 and the uniqueness schema in Sources/UttrflowPredictStore/Schema.swift:38. Check Surroundings.cleaned at Sources/UttrflowContext/Surroundings.swift:197-203, then use the named migration and prediction tests in Tests/UttrflowPredictStoreTests/PredictStoreTests.swift or CaseMatchingTests.swift. Done means NFC forms share one merged row, match through both candidate paths, and ZWJ/ZWNJ remain in cleaned context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sqlite, swift
- Domain
- databases, desktop
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100