uttrflow / uttrflow/uttrflow-swift
uttrflow-bakeoff --models silently measures nothing when given a catalogue name such as gemma3Small
- Dominant language
- Swift
- Stars
- 4
- Forks
- 17
- Avg merge
- 3h 32m
- Merged PRs (30d)
- 277
Description
## What happens
`Sources/uttrflow-bakeoff/Bakeoff.swift:103-107` turns `--models` into models with `compactMap { LocalModel.named(...) }`. `LocalModel.named` (`Sources/UttrflowLocalModel/LocalModel.swift:111-113`) only matches the repository identifier (`mlx-community/gemma-3-1b-it-qat-4bit`) or its last path component (`gemma-3-1b-it-qat-4bit`). The Swift names people see in the catalogue, such as `gemma3Small` or `qwen3`, match nothing and are dropped without a word.
Because `--models` was given, line 62 also skips the baselines. So `uttrflow-bakeoff --models gemma3Small` measures nothing at all and then prints the old results as if it had run.
## Why it matters
A bake-off run that quietly does nothing looks like a run that found nothing new. It can waste a long measuring session.
## How to check
Read `selectedModels()` and `LocalModel.named`. `LocalModel.named("gemma3Small")` returns `nil` today; a one-line test in `Tests/UttrflowLocalModelTests/LocalModelTests.swift` shows it.
## Acceptance criteria
- Either `LocalModel.named` also accepts the catalogue names (`gemma3Small`, `llama32`, `qwen3`, `ministral3`, `gemma3`), or the bake-off refuses an unknown name. Preferably both: accept the catalogue names, and fail with a message listing the valid names for anything else.
- An unknown name in `--models` never results in a run that measures nothing without saying so.
- Tests in `LocalModelTests.swift` cover every accepted spelling and an unknown one.
## Where to start
- `Sources/UttrflowLocalModel/LocalModel.swift:111-113`
- `Sources/uttrflow-bakeoff/Bakeoff.swift:103-107`
- Tests to extend: `Tests/UttrflowLocalModelTests/LocalModelTests.swift`.
- Before pushing, run `make verify` (export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer first). It is the same command CI runs, and it enforces the 95% coverage floor per module.
- Read [CONTRIBUTING.md](https://github.com/uttrflow/uttrflow-swift/blob/main/CONTRIBUTING.md) first, and say on this issue that you are taking it.
**Size:** S, about 2 hours.
Contributor guide
Research direction
Read CONTRIBUTING.md, then inspect selectedModels() in Sources/uttrflow-bakeoff/Bakeoff.swift and LocalModel.named in Sources/UttrflowLocalModel/LocalModel.swift. Extend Tests/UttrflowLocalModelTests/LocalModelTests.swift for the catalogue spellings and an unknown name, and verify that an invalid --models value is reported rather than producing an empty run. Run make verify with the specified DEVELOPER_DIR before pushing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- cli, testing
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100