uttrflow / uttrflow/uttrflow-swift

The clipboard oracle suites take 76 s of a 95 s local test run and 233 s on CI, and the tests starved behind them have failed main

Closed Beginner friendly
#632 0 comments 0 reactions 0 assignees View on GitHub
area:clipboard bug good first issue P1 performance
Dominant language
Swift
Stars
4
Forks
17
Avg merge
4h 27m
Merged PRs (30d)
286

Description

## What happens

Two suites added with the linear-time clipboard readers (#443, #460) compare the new readers against the old backtracking patterns on hundreds of thousands of generated strings:

- `Tests/UttrflowClipboardTests/SecretShapesOracleTests.swift:9` — "The linear readers answer exactly as the backtracking patterns did". `randomStrings` (line 96) runs 8 seeds × 25,000 strings; `plantedSecrets` (line 113) 4 × 10,000.
- `Tests/UttrflowClipboardTests/ClipKindOracleTests.swift:9` — "The windowed classifier answers as the whole-clip classifier did". Lines 66, 74, 82 run 4 + 2 + 4 seeds × 5,000 strings each, and line 166 another 20,000.

Both suites are `.serialized`, so the seeds run one after another, and each case runs on its own thread through `offTheTestPool` (`SecretShapesOracleTests.swift:195`), which keeps the cores busy while every other suite runs.

Measured on origin/main (26d7bc1), Apple Silicon, `swift test --parallel`:

| Run | Wall time |
|---|---|
| Whole suite | 96 s, 5,384 tests |
| `--filter 'SecretShapesOracleTests\|ClipKindOracleTests'` (13 tests) | 76 s |
| `--skip` those two suites (5,371 tests) | 18.6 s |

On the macOS CI runner, under `make verify`'s coverage build, the two suites take **233 s** and the whole test run ends when they do ([run 34815318669](https://github.com/uttrflow/uttrflow-swift/actions/runs/34815318669): "Test run with 5366 tests in 726 suites failed after 233.632 seconds").

While they run, unrelated tests stall. In that run, at the second both oracle suites started (07:08:48), dozens of small tests began and then "passed after 29.8–33.5 seconds", and four tests in `DictationPipelineTurnTests` hit that file's 30-second wall-clock wait (`DictationPipelineTurnTests.swift:150`) and failed. The same four-to-seven failures at the same line broke CI on `main` twice on 2026-09-14 (runs 34815318669 and 34796754962) and once on a pull request branch (34792317387). The 30-second waits themselves are tracked in #629; this issue is the load that trips them.

## Why it matters

Every contributor waits for these 13 tests on every `make test` and `make verify`, locally about four times as long as the other 5,371 tests together. On CI they make `main` go red for changes that touched nothing near the clipboard, which is exactly the failure CONTRIBUTING.md promises cannot happen.

## How to reproduce

```bash
swift build --build-tests
swift test --skip-build --filter 'SecretShapesOracleTests|ClipKindOracleTests'
swift test --skip-build --skip 'SecretShapesOracleTests|ClipKindOracleTests'
```

## Acceptance criteria

- By default, the two suites together take under 5 s of wall time on an Apple Silicon Mac (for example 2 seeds × 2,000 strings for the random-text cases, keeping every planted-shape and hand-written case as it is).
- The full sweep is still available behind an environment variable, following `UTTRFLOW_TOKENIZER_PROOF`, e.g. `UTTRFLOW_ORACLE_SWEEP=1 swift test --filter Oracle`, and the variable is named in a comment at the top of each file.
- Every generator still runs by default (at least seed 0 of each), so a regression in the readers still fails `make verify`.
- `make verify` still passes, including the 95% coverage floor for `UttrflowClipboard`.

## Where to start

- `Tests/UttrflowClipboardTests/SecretShapesOracleTests.swift` and `Tests/UttrflowClipboardTests/ClipKindOracleTests.swift` are the only files to change.
- `Tests/UttrflowLocalModelTests/PromptTokensTests.swift:321-334` shows how `UTTRFLOW_TOKENIZER_PROOF` switches a slow test on only when it is set.
- Read [CONTRIBUTING.md](https://github.com/uttrflow/uttrflow-swift/blob/main/CONTRIBUTING.md) first, and say on this issue that you are taking it.
- Before pushing, run `make verify` (with `DEVELOPER_DIR` pointing at Xcode 26.6 or later). It is the same command CI runs.

**Size:** S, about 1–2 hours.

Contributor guide

Open the contributing guide

Research direction

Read CONTRIBUTING.md first, then run the two filter commands from the issue to establish the baseline. Inspect Tests/UttrflowClipboardTests/SecretShapesOracleTests.swift and ClipKindOracleTests.swift, using PromptTokensTests.swift:321-334 as the environment-variable example. Done means the default suites meet the stated time and coverage requirements, the full sweep is opt-in, and make verify passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
performance, testing
Issue type
Refactor
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.