uttrflow / uttrflow/uttrflow-swift

CRLF model output is flattened by TextTidy and valid multiline cleanup is rejected

Open
#814 0 comments 0 reactions 1 assignee Claimed by @naveen-bhatt View on GitHub
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

## Problem

`TextTidy.collapseSpacing` promises to preserve a model answer's line breaks, but only splits on the LF `Character`. Swift treats CRLF as one character, so CRLF remains inside a chunk and the subsequent whitespace splitting converts it to a space.

On main `5c310667cd35c44abd8bbd26c19b0905471287b4`, equivalent LF and CRLF model responses take different paths: the LF response is accepted, while the CRLF response is altered by cleanup and then rejected for dropping a requested line break.

## Reproduction and evidence

Use the actual `GenerativeTextTransformer` with the repository's scripted `FakeCleanupModel` and a request for:

```text
do we discount new line answer colon we do not discount
```

Have the model return `Do we discount?` followed by the tested newline and `Answer: We do not discount.`. Run the same request through `TransformerRouter` with this transformer followed by `RuleBasedTransformer`.

Observed with the production helpers and engines:

```text
LF:
unwrapped="Do we discount?\nAnswer: We do not discount."
tidied="Do we discount?\nAnswer: We do not discount."
router=foundationModels
result="Do we discount?\nAnswer: We do not discount."

CRLF:
unwrapped="Do we discount?\r\nAnswer: We do not discount."
tidied="Do we discount? Answer: We do not discount."
direct transformer throws outputRejected:
"the rewrite dropped a line break the speaker asked for"
router=rules
result="Do we discount\nanswer: we do not discount."
```

The LF control passes. The CRLF acceptance assertion fails with the rejection above. The 60 existing transformation, rules, spacing, script-guard, Latin-only and warming tests passed in the isolated harness before this probe.

This uses a controlled model response, not a captured Apple Foundation Models response. It proves handling of a valid newline representation at the `CleanupModel` boundary; it does not establish how frequently the current model emits CRLF.

## Location

- [TextTidy.swift](https://github.com/uttrflow/uttrflow-swift/blob/5c310667cd35c44abd8bbd26c19b0905471287b4/Sources/UttrflowAI/TextTidy.swift#L13-L20): LF-only split followed by whitespace collapse.
- [GenerativeTextTransformer.swift](https://github.com/uttrflow/uttrflow-swift/blob/5c310667cd35c44abd8bbd26c19b0905471287b4/Sources/UttrflowAI/GenerativeTextTransformer.swift#L59-L80): unwrapping, spacing cleanup, finishing and guard validation.
- `TextTidyTests.collapseSpacing` tests LF and blank lines, but no CRLF case.

`ResponseUnwrapper` preserves CRLF for this answer; the first observed loss is the spacing helper.

## Impact and acceptance criteria

Priority P2. Correctly laid-out model output is unnecessarily discarded, and the user receives the less polished rules result. The guard prevents silent insertion of the flattened result in this reproduction.

- Treat LF and CRLF consistently when preserving line breaks, including consecutive blank lines and mixed endings.
- Preserve the helper's intended horizontal-whitespace cleanup.
- Add helper and full-transformer regression coverage showing equivalent accepted output for the two model responses.
- Keep the rejection for a model answer that actually omits a requested break.

This is separate from #803, which concerns clipboard formatter-diff line splitting, and #789, which concerns context supplied to suggestions. Neither fixes this dictation-cleanup path.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.