uttrflow / uttrflow/uttrflow-swift
Two grammar evaluation cases pass outputs that leave the targeted grammar error unchanged
- Dominant language
- Swift
- Stars
- 4
- Forks
- 17
- Avg merge
- 4h 27m
- Merged PRs (30d)
- 286
Description
## Problem
The grammar cases `agreement-he-dont` and `preposition-slip` do not require the repair they exist to measure. Their required words and beginning/end checks are satisfied by the original uncorrected sentence; its token overlap also exceeds `CaseScore.passed`'s 0.8 threshold.
## Reproduction
On main `5c310667cd35c44abd8bbd26c19b0905471287b4`, call the unchanged production `Scorer.score` with these existing `EvaluationCorpus` cases:
| Case | Output supplied to scorer | Expected repair | Actual result |
| --- | --- | --- | --- |
| `agreement-he-dont` | `He don't know about the meeting yet.` | `He doesn't know about the meeting yet.` | passes, similarity 0.875 |
| `preposition-slip` | `She is good in maths and physics.` | `She is good at maths and physics.` | passes, similarity approximately 0.8571 |
Both results have empty `lost`, `invented`, and `brokeShape` arrays. A focused Swift test expecting these unrepaired outputs to fail records two failures. As a control, the existing `tense-drift` case correctly rejects its uncorrected output (similarity approximately 0.7778).
The probe checks scoring logic with controlled outputs; it does not claim that a real model produced these answers. `EvaluationReport.passRate` counts these scores as passes, so the reported result can credit a model for a repair it did not perform.
## Acceptance criteria
- Require the targeted repair in each repair-only grammar case, using case-specific guards or an explicit comparison appropriate to the case.
- Add negative controls that submit each original unrepaired sentence and require failure, alongside positive controls for the corrected reference.
- Keep deliberate dialect-preservation and messaging/as-spoken cases distinct; their unchanged grammar is intentional.
- Avoid globally tightening the similarity threshold as a substitute for asserting the specific behavior.
Relevant code: `Sources/UttrflowEval/EvaluationCorpus.swift` (grammar cases), `Sources/UttrflowEval/CaseScore.swift` (`passed`, report `passRate`), and the existing corpus scoring tests.
Contributor guide
Research direction
Start with the grammar cases in Sources/UttrflowEval/EvaluationCorpus.swift and the passed logic in Sources/UttrflowEval/CaseScore.swift. Read the existing corpus scoring tests, then add negative controls for the two unrepaired sentences and positive controls for their corrected references. Done means each targeted repair is required without changing intentional dialect-preservation or messaging cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100