uttrflow / uttrflow/uttrflow-swift
The clipboard oracle suites take 76 s of a 95 s local test run and 233 s on CI, and the tests starved behind them have failed main
- Dominant language
- Swift
- Stars
- 4
- Forks
- 17
- Avg merge
- 4h 27m
- Merged PRs (30d)
- 286
Description
## What happens
Two suites added with the linear-time clipboard readers (#443, #460) compare the new readers against the old backtracking patterns on hundreds of thousands of generated strings:
- `Tests/UttrflowClipboardTests/SecretShapesOracleTests.swift:9` — "The linear readers answer exactly as the backtracking patterns did". `randomStrings` (line 96) runs 8 seeds × 25,000 strings; `plantedSecrets` (line 113) 4 × 10,000.
- `Tests/UttrflowClipboardTests/ClipKindOracleTests.swift:9` — "The windowed classifier answers as the whole-clip classifier did". Lines 66, 74, 82 run 4 + 2 + 4 seeds × 5,000 strings each, and line 166 another 20,000.
Both suites are `.serialized`, so the seeds run one after another, and each case runs on its own thread through `offTheTestPool` (`SecretShapesOracleTests.swift:195`), which keeps the cores busy while every other suite runs.
Measured on origin/main (26d7bc1), Apple Silicon, `swift test --parallel`:
| Run | Wall time |
|---|---|
| Whole suite | 96 s, 5,384 tests |
| `--filter 'SecretShapesOracleTests\|ClipKindOracleTests'` (13 tests) | 76 s |
| `--skip` those two suites (5,371 tests) | 18.6 s |
On the macOS CI runner, under `make verify`'s coverage build, the two suites take **233 s** and the whole test run ends when they do ([run 34815318669](https://github.com/uttrflow/uttrflow-swift/actions/runs/34815318669): "Test run with 5366 tests in 726 suites failed after 233.632 seconds").
While they run, unrelated tests stall. In that run, at the second both oracle suites started (07:08:48), dozens of small tests began and then "passed after 29.8–33.5 seconds", and four tests in `DictationPipelineTurnTests` hit that file's 30-second wall-clock wait (`DictationPipelineTurnTests.swift:150`) and failed. The same four-to-seven failures at the same line broke CI on `main` twice on 2026-09-14 (runs 34815318669 and 34796754962) and once on a pull request branch (34792317387). The 30-second waits themselves are tracked in #629; this issue is the load that trips them.
## Why it matters
Every contributor waits for these 13 tests on every `make test` and `make verify`, locally about four times as long as the other 5,371 tests together. On CI they make `main` go red for changes that touched nothing near the clipboard, which is exactly the failure CONTRIBUTING.md promises cannot happen.
## How to reproduce
```bash
swift build --build-tests
swift test --skip-build --filter 'SecretShapesOracleTests|ClipKindOracleTests'
swift test --skip-build --skip 'SecretShapesOracleTests|ClipKindOracleTests'
```
## Acceptance criteria
- By default, the two suites together take under 5 s of wall time on an Apple Silicon Mac (for example 2 seeds × 2,000 strings for the random-text cases, keeping every planted-shape and hand-written case as it is).
- The full sweep is still available behind an environment variable, following `UTTRFLOW_TOKENIZER_PROOF`, e.g. `UTTRFLOW_ORACLE_SWEEP=1 swift test --filter Oracle`, and the variable is named in a comment at the top of each file.
- Every generator still runs by default (at least seed 0 of each), so a regression in the readers still fails `make verify`.
- `make verify` still passes, including the 95% coverage floor for `UttrflowClipboard`.
## Where to start
- `Tests/UttrflowClipboardTests/SecretShapesOracleTests.swift` and `Tests/UttrflowClipboardTests/ClipKindOracleTests.swift` are the only files to change.
- `Tests/UttrflowLocalModelTests/PromptTokensTests.swift:321-334` shows how `UTTRFLOW_TOKENIZER_PROOF` switches a slow test on only when it is set.
- Read [CONTRIBUTING.md](https://github.com/uttrflow/uttrflow-swift/blob/main/CONTRIBUTING.md) first, and say on this issue that you are taking it.
- Before pushing, run `make verify` (with `DEVELOPER_DIR` pointing at Xcode 26.6 or later). It is the same command CI runs.
**Size:** S, about 1–2 hours.
Contributor guide
Research direction
Read CONTRIBUTING.md first, then run the two filter commands from the issue to establish the baseline. Inspect Tests/UttrflowClipboardTests/SecretShapesOracleTests.swift and ClipKindOracleTests.swift, using PromptTokensTests.swift:321-334 as the environment-variable example. Done means the default suites meet the stated time and coverage requirements, the full sweep is opt-in, and make verify passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- performance, testing
- Issue type
- Refactor
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100