uttrflow / uttrflow/uttrflow-swift

The suggestion store never normalises Unicode, so the same Hindi or accented line typed through two input sources is stored twice and one spelling can't find the other

Open
#717 1 comment 0 reactions 0 assignees View on GitHub
area:ai-suggestions bug P2
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

## What happens

No code under `Sources/` normalises text: there is no `precomposedStringWithCanonicalMapping`, `decomposedStringWithCanonicalMapping` or equivalent. Swift's `String ==` compares canonically equivalent strings as equal, but SQLite compares bytes. The suggestion store works in SQL:

- the prefix scan: `text_lower >= ? AND text_lower < ?` (`Sources/UttrflowPredictStore/PredictStore.swift:165-186`);
- uniqueness: `UNIQUE (surface_id, text)` (`Sources/UttrflowPredictStore/Schema.swift:38`);
- the fuzzy fallback, over UTF-8 bytes (`PredictStore.swift:189-215`).

Devanagari nukta letters have two encodings: ज़ is U+095B, or ज U+091C + ़ U+093C. Latin accents do too: é is U+00E9, or e + U+0301. Which one arrives depends on the keyboard layout, the input method and where text was pasted from. Checked with the `sqlite3` tool:

```
create table e(t text, unique(t));
insert into e values(char(0x95B)||'िंदगी'); -- ज़िंदगी, precomposed
insert into e values(char(0x91C,0x93C)||'िंदगी'); -- ज़िंदगी, decomposed: accepted, 2 rows
select count(*) from e where t >= char(0x91C,0x93C) and t < char(0x91C,0x93D); -- 1
```

In Swift the two strings are `==`, but their UTF-8 differs. So:

- the same line is learned twice, and its count and recency are split between the two rows;
- a prefix typed in one form doesn't find the line stored in the other.

`Surroundings.cleaned` (`Sources/UttrflowContext/Surroundings.swift:197-203`) strips every `.format` scalar from screen context. That includes ZWJ and ZWNJ (U+200D, U+200C), which Hindi typing uses to pick a half-form. Stored lines keep them. Only the prompt context is affected, which is minor.

## Why it matters

Hindi users commonly switch between a phonetic input method and a keyboard layout, and paste from messaging apps. Each can produce a different encoding of the same word, most visibly with nukta letters (ज़, फ़, क़, ड़ are everyday letters). Their suggestions then learn more slowly and miss matches an English user would get.

## Acceptance criteria

- Text is normalised to NFC in one place, before it is stored and before it is matched: the store's `learn`/record paths and the typed prefix in `exactCandidates`/`fuzzyCandidates`.
- Existing rows are normalised once by a schema migration. Rows that become duplicates are merged, adding their counts.
- ZWJ and ZWNJ are kept when context is cleaned (strip other `.format` scalars as today).
- Tests in `Tests/UttrflowPredictStoreTests/PredictStoreTests.swift` (or `CaseMatchingTests.swift`): the precomposed and decomposed forms of "ज़िंदगी" and "café" are one entry and find each other.

Contributor guide

Open the contributing guide

Research direction

Start with the SQL paths in Sources/UttrflowPredictStore/PredictStore.swift:165-215 and the uniqueness schema in Sources/UttrflowPredictStore/Schema.swift:38. Check Surroundings.cleaned at Sources/UttrflowContext/Surroundings.swift:197-203, then use the named migration and prediction tests in Tests/UttrflowPredictStoreTests/PredictStoreTests.swift or CaseMatchingTests.swift. Done means NFC forms share one merged row, match through both candidate paths, and ZWJ/ZWNJ remain in cleaned context.

Written by the indexing model from the issue text.

Assessment

Tech stack
sqlite, swift
Domain
databases, desktop
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.