uttrflow / uttrflow/uttrflow-swift
The suggestion store measures a line with Swift's character count but SQLite counts code points, so half-typed Hindi (and emoji) lines are stored as fragments and never retired
- Dominant language
- Swift
- Stars
- 4
- Forks
- 17
- Avg merge
- 3h 32m
- Merged PRs (30d)
- 277
Description
## What happens
`PredictStore` keeps its learned lines free of fragments with two SQL checks. Both bind a Swift `String.count`, which counts grapheme clusters, into SQLite's `length()` and `substr()`, which count code points.
- `isFragmentOfLongerEntry` (`Sources/UttrflowPredictStore/PredictStore.swift:321-341`) binds `Int64(lowered.count)` into `length(text_lower) > ? AND substr(text_lower, 1, ?) = ?`.
- `supersedeFragments` (`:344-362`) binds `Int64(lowered.count)` into `length(text_lower) < ?`.
For ASCII the two counts are equal. For Devanagari they aren't: "नहीं" is 2 Characters but 4 code points, and "नहीं जाना" is 5 Characters but 9 code points.
```
sqlite> select length('नहीं'), length('नहीं जाना'), substr('नहीं जाना', 1, 2);
4|9|नह
```
So, with "नहीं जाना" stored:
- `isFragmentOfLongerEntry("नहीं")` compares `substr(text_lower, 1, 2)` = "नह" with "नहीं" and finds nothing. The fragment "नहीं" is stored.
- `supersedeFragments("नहीं जाना")` asks for entries with `length < 5`, so "नहीं जा" (7 code points) is never retired.
Checked with the `sqlite3` command-line tool and the exact SQL on `origin/main` 26d7bc1. Emoji with modifiers and ZWJ sequences are affected the same way.
## Why it matters
Half-typed Hindi lines stay live next to the whole line. They compete with it in ranking and narrow the gap between the top two suggestions, which decides whether anything is shown. The store's own tests say this shouldn't happen ("A longer line retires the fragments it grew out of, so only the whole value is offered.", `PredictStoreTests.swift:74`), but they use Latin text only.
## Acceptance criteria
- Both queries bind a length in the unit SQLite counts: `lowered.unicodeScalars.count`. Alternatively, compare with a byte-safe prefix test (`text_lower >= ? AND text_lower < upperBound`, the way `prefixQuery` already does) instead of `substr`/`length`.
- Tests in `Tests/UttrflowPredictStoreTests/PredictStoreTests.swift`:
- after learning "नहीं जाना", learning "नहीं" doesn't create an entry;
- learning "नहीं जा" and then "नहीं जाना" leaves "नहीं जा" superseded;
- the same two cases with an emoji line such as "ok 👍🏽 done".
- Existing Latin fragment tests still pass.
## Where to start
- `Sources/UttrflowPredictStore/PredictStore.swift:321-362`, and `prefixQuery`/`upperBound` around `:165-186` for the byte-ordered alternative
- Test to extend: `Tests/UttrflowPredictStoreTests/PredictStoreTests.swift` (the fragment tests at `:64` and `:74`)
- Before pushing, run `make verify` (export `DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer` first). It is the same command CI runs, and it enforces the 95% coverage floor per module.
- Read [CONTRIBUTING.md](https://github.com/uttrflow/uttrflow-swift/blob/main/CONTRIBUTING.md) first, and say on this issue that you are taking it.
**Size:** S, about 2 hours.
Contributor guide
Research direction
Read CONTRIBUTING.md, then inspect isFragmentOfLongerEntry and supersedeFragments in Sources/UttrflowPredictStore/PredictStore.swift:321-362, along with prefixQuery and upperBound around :165-186. Extend Tests/UttrflowPredictStoreTests/PredictStoreTests.swift at the fragment tests near :64 and :74 for Devanagari and emoji cases, preserving the existing Latin coverage. Run make verify with DEVELOPER_DIR set; done means all acceptance cases pass and CI coverage remains satisfied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sqlite, swift
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 86/100