uttrflow / uttrflow/uttrflow-swift

The suggestion store measures a line with Swift's character count but SQLite counts code points, so half-typed Hindi (and emoji) lines are stored as fragments and never retired

Open Beginner friendly
#715 1 comment 0 reactions 0 assignees View on GitHub
area:ai-suggestions bug good first issue P2
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

## What happens

`PredictStore` keeps its learned lines free of fragments with two SQL checks. Both bind a Swift `String.count`, which counts grapheme clusters, into SQLite's `length()` and `substr()`, which count code points.

- `isFragmentOfLongerEntry` (`Sources/UttrflowPredictStore/PredictStore.swift:321-341`) binds `Int64(lowered.count)` into `length(text_lower) > ? AND substr(text_lower, 1, ?) = ?`.
- `supersedeFragments` (`:344-362`) binds `Int64(lowered.count)` into `length(text_lower) < ?`.

For ASCII the two counts are equal. For Devanagari they aren't: "नहीं" is 2 Characters but 4 code points, and "नहीं जाना" is 5 Characters but 9 code points.

```
sqlite> select length('नहीं'), length('नहीं जाना'), substr('नहीं जाना', 1, 2);
4|9|नह
```

So, with "नहीं जाना" stored:

- `isFragmentOfLongerEntry("नहीं")` compares `substr(text_lower, 1, 2)` = "नह" with "नहीं" and finds nothing. The fragment "नहीं" is stored.
- `supersedeFragments("नहीं जाना")` asks for entries with `length < 5`, so "नहीं जा" (7 code points) is never retired.

Checked with the `sqlite3` command-line tool and the exact SQL on `origin/main` 26d7bc1. Emoji with modifiers and ZWJ sequences are affected the same way.

## Why it matters

Half-typed Hindi lines stay live next to the whole line. They compete with it in ranking and narrow the gap between the top two suggestions, which decides whether anything is shown. The store's own tests say this shouldn't happen ("A longer line retires the fragments it grew out of, so only the whole value is offered.", `PredictStoreTests.swift:74`), but they use Latin text only.

## Acceptance criteria

- Both queries bind a length in the unit SQLite counts: `lowered.unicodeScalars.count`. Alternatively, compare with a byte-safe prefix test (`text_lower >= ? AND text_lower < upperBound`, the way `prefixQuery` already does) instead of `substr`/`length`.
- Tests in `Tests/UttrflowPredictStoreTests/PredictStoreTests.swift`:
- after learning "नहीं जाना", learning "नहीं" doesn't create an entry;
- learning "नहीं जा" and then "नहीं जाना" leaves "नहीं जा" superseded;
- the same two cases with an emoji line such as "ok 👍🏽 done".
- Existing Latin fragment tests still pass.

## Where to start

- `Sources/UttrflowPredictStore/PredictStore.swift:321-362`, and `prefixQuery`/`upperBound` around `:165-186` for the byte-ordered alternative
- Test to extend: `Tests/UttrflowPredictStoreTests/PredictStoreTests.swift` (the fragment tests at `:64` and `:74`)
- Before pushing, run `make verify` (export `DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer` first). It is the same command CI runs, and it enforces the 95% coverage floor per module.
- Read [CONTRIBUTING.md](https://github.com/uttrflow/uttrflow-swift/blob/main/CONTRIBUTING.md) first, and say on this issue that you are taking it.

**Size:** S, about 2 hours.

Contributor guide

Open the contributing guide

Research direction

Read CONTRIBUTING.md, then inspect isFragmentOfLongerEntry and supersedeFragments in Sources/UttrflowPredictStore/PredictStore.swift:321-362, along with prefixQuery and upperBound around :165-186. Extend Tests/UttrflowPredictStoreTests/PredictStoreTests.swift at the fragment tests near :64 and :74 for Devanagari and emoji cases, preserving the existing Latin coverage. Run make verify with DEVELOPER_DIR set; done means all acceptance cases pass and CI coverage remains satisfied.

Written by the indexing model from the issue text.

Assessment

Tech stack
sqlite, swift
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
86/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.