uttrflow / uttrflow/uttrflow-swift

Prompt alignment copies IOSurface row padding into dense weights, corrupting word timestamps

Open
#787 0 comments 0 reactions 0 assignees View on GitHub
area:dictation bug confirmed P2
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

## Problem

`PromptAlignedSegmentSeeker.rows(of:from:)` shifts alignment weights with one bulk byte copy. The source and destination do not have the same row stride on the current WhisperKit path, so the copy carries source padding into destination data. Later alignment rows no longer represent the intended tokens.

This affects prompted decodes, where the app installs this seeker. Incorrect word timestamps also feed the upstream segment/seek calculations. Timestamp corruption is reproduced below; no real-audio sentence-loss rate or crash is claimed.

## Evidence

App main: `8d93f5b8574f3c7b8c30086d9c817bf7f5d4827d`. Linked dependency: clean WhisperKit v1.1.0, `1e2a163736dfa5a198e637ae44c114e1c6d5cc2d`.

- `Sources/UttrflowSpeech/PromptAlignedSegmentSeeker.swift:54-67` allocates a destination with `MLMultiArray(shape:dataType:)`, derives `rowBytes` from the source byte count, then copies all retained rows together.
- WhisperKit `Sources/WhisperKit/Core/TextDecoder.swift:141` allocates its float16 alignment buffer using the `initialValue:` initializer. `Sources/ArgmaxCore/MLMultiArrayExtensions.swift:11-18` implements that float16 initializer with an IOSurface-backed pixel buffer.
- `TextDecoder.swift:784` and `TranscribeTask.swift:198-203` pass this buffer to the segment seeker.

Release-mode reproduction using the exact upstream initializer and unchanged production row-shift code:

| buffer, shape 224 × 1500 | strides | exposed bytes |
|---|---|---:|
| upstream float16 source | `[1504, 1]` | 673792 |
| ordinary destination | `[1500, 1]` | 672000 |

After filling each source row's first element with its row index and shifting by four rows, destination `[219, 0]` is **0**, where the expected source `[223, 0]` is **223**.

A second reproduction uses the existing synthetic alignment fixture and the real WhisperKit `SegmentSeeker`, changing the prompted buffer allocation to the upstream IOSurface initializer and populating it by logical indices. The first word's expected interval is **0.12–0.16 s**; the prompted result is **0.72–0.96 s**. All five word intervals differ. The equivalent existing test with ordinary dense allocation passes.

## Fix and acceptance criteria

Copy logical alignment elements or rows using the actual source and destination strides. Preserve shape and data type, and zero the trailing logical rows. Do not copy padding as data or assume equal physical row sizes merely because shapes match.

Add a regression using the upstream float16 initializer, including a width with padded rows such as 1500. Verify every retained logical row and the zeroed tail. Extend the real-seeker equivalence test to cover this allocation, retaining the ordinary dense-buffer case. Validate a prompted audio decode after the fix to assess its effect on segment retention and seek positions.

Related: closed #423 introduced prompt-offset alignment. This is a distinct defect in the resulting row-copy implementation: the offset is applied, but unequal physical strides corrupt the shifted matrix. Existing tests allocate dense buffers and miss this boundary.

Priority: P2. Two release-mode reproductions fail their correctness assertions; production source is unchanged.

Contributor guide

Open the contributing guide

Research direction

Start in Sources/UttrflowSpeech/PromptAlignedSegmentSeeker.swift:54-67 and compare its row copy with the source and destination strides. Run the existing synthetic alignment and real-seeker equivalence tests, then add coverage using WhisperKit’s float16 initializer with a padded width such as 1500. Done means retained logical rows match and the tail is zeroed without copying padding, with the prompted decode behavior checked afterward.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, swift
Domain
audio-video-rtc, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.