uttrflow / uttrflow/uttrflow-swift

uttrflow-bakeoff --models silently measures nothing when given a catalogue name such as gemma3Small

Open Beginner friendly
#494 0 comments 0 reactions 0 assignees View on GitHub
area:ai-suggestions area:build bug good first issue P2
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

## What happens

`Sources/uttrflow-bakeoff/Bakeoff.swift:103-107` turns `--models` into models with `compactMap { LocalModel.named(...) }`. `LocalModel.named` (`Sources/UttrflowLocalModel/LocalModel.swift:111-113`) only matches the repository identifier (`mlx-community/gemma-3-1b-it-qat-4bit`) or its last path component (`gemma-3-1b-it-qat-4bit`). The Swift names people see in the catalogue, such as `gemma3Small` or `qwen3`, match nothing and are dropped without a word.

Because `--models` was given, line 62 also skips the baselines. So `uttrflow-bakeoff --models gemma3Small` measures nothing at all and then prints the old results as if it had run.

## Why it matters

A bake-off run that quietly does nothing looks like a run that found nothing new. It can waste a long measuring session.

## How to check

Read `selectedModels()` and `LocalModel.named`. `LocalModel.named("gemma3Small")` returns `nil` today; a one-line test in `Tests/UttrflowLocalModelTests/LocalModelTests.swift` shows it.

## Acceptance criteria

- Either `LocalModel.named` also accepts the catalogue names (`gemma3Small`, `llama32`, `qwen3`, `ministral3`, `gemma3`), or the bake-off refuses an unknown name. Preferably both: accept the catalogue names, and fail with a message listing the valid names for anything else.
- An unknown name in `--models` never results in a run that measures nothing without saying so.
- Tests in `LocalModelTests.swift` cover every accepted spelling and an unknown one.

## Where to start

- `Sources/UttrflowLocalModel/LocalModel.swift:111-113`
- `Sources/uttrflow-bakeoff/Bakeoff.swift:103-107`
- Tests to extend: `Tests/UttrflowLocalModelTests/LocalModelTests.swift`.
- Before pushing, run `make verify` (export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer first). It is the same command CI runs, and it enforces the 95% coverage floor per module.
- Read [CONTRIBUTING.md](https://github.com/uttrflow/uttrflow-swift/blob/main/CONTRIBUTING.md) first, and say on this issue that you are taking it.

**Size:** S, about 2 hours.

Contributor guide

Open the contributing guide

Research direction

Read CONTRIBUTING.md, then inspect selectedModels() in Sources/uttrflow-bakeoff/Bakeoff.swift and LocalModel.named in Sources/UttrflowLocalModel/LocalModel.swift. Extend Tests/UttrflowLocalModelTests/LocalModelTests.swift for the catalogue spellings and an unknown name, and verify that an invalid --models value is reported rather than producing an empty run. Run make verify with the specified DEVELOPER_DIR before pushing.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
cli, testing
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.