uttrflow / uttrflow/uttrflow-swift

Two grammar evaluation cases pass outputs that leave the targeted grammar error unchanged

Open
#826 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
4
Forks
17
Avg merge
4h 27m
Merged PRs (30d)
286

Description

## Problem

The grammar cases `agreement-he-dont` and `preposition-slip` do not require the repair they exist to measure. Their required words and beginning/end checks are satisfied by the original uncorrected sentence; its token overlap also exceeds `CaseScore.passed`'s 0.8 threshold.

## Reproduction

On main `5c310667cd35c44abd8bbd26c19b0905471287b4`, call the unchanged production `Scorer.score` with these existing `EvaluationCorpus` cases:

| Case | Output supplied to scorer | Expected repair | Actual result |
| --- | --- | --- | --- |
| `agreement-he-dont` | `He don't know about the meeting yet.` | `He doesn't know about the meeting yet.` | passes, similarity 0.875 |
| `preposition-slip` | `She is good in maths and physics.` | `She is good at maths and physics.` | passes, similarity approximately 0.8571 |

Both results have empty `lost`, `invented`, and `brokeShape` arrays. A focused Swift test expecting these unrepaired outputs to fail records two failures. As a control, the existing `tense-drift` case correctly rejects its uncorrected output (similarity approximately 0.7778).

The probe checks scoring logic with controlled outputs; it does not claim that a real model produced these answers. `EvaluationReport.passRate` counts these scores as passes, so the reported result can credit a model for a repair it did not perform.

## Acceptance criteria

- Require the targeted repair in each repair-only grammar case, using case-specific guards or an explicit comparison appropriate to the case.
- Add negative controls that submit each original unrepaired sentence and require failure, alongside positive controls for the corrected reference.
- Keep deliberate dialect-preservation and messaging/as-spoken cases distinct; their unchanged grammar is intentional.
- Avoid globally tightening the similarity threshold as a substitute for asserting the specific behavior.

Relevant code: `Sources/UttrflowEval/EvaluationCorpus.swift` (grammar cases), `Sources/UttrflowEval/CaseScore.swift` (`passed`, report `passRate`), and the existing corpus scoring tests.

Contributor guide

Open the contributing guide

Research direction

Start with the grammar cases in Sources/UttrflowEval/EvaluationCorpus.swift and the passed logic in Sources/UttrflowEval/CaseScore.swift. Read the existing corpus scoring tests, then add negative controls for the two unrepaired sentences and positive controls for their corrected references. Done means each targeted repair is required without changing intentional dialect-preservation or messaging cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.