uttrflow / uttrflow/uttrflow-swift
Prompt alignment copies IOSurface row padding into dense weights, corrupting word timestamps
- Dominant language
- Swift
- Stars
- 4
- Forks
- 17
- Avg merge
- 3h 32m
- Merged PRs (30d)
- 277
Description
## Problem
`PromptAlignedSegmentSeeker.rows(of:from:)` shifts alignment weights with one bulk byte copy. The source and destination do not have the same row stride on the current WhisperKit path, so the copy carries source padding into destination data. Later alignment rows no longer represent the intended tokens.
This affects prompted decodes, where the app installs this seeker. Incorrect word timestamps also feed the upstream segment/seek calculations. Timestamp corruption is reproduced below; no real-audio sentence-loss rate or crash is claimed.
## Evidence
App main: `8d93f5b8574f3c7b8c30086d9c817bf7f5d4827d`. Linked dependency: clean WhisperKit v1.1.0, `1e2a163736dfa5a198e637ae44c114e1c6d5cc2d`.
- `Sources/UttrflowSpeech/PromptAlignedSegmentSeeker.swift:54-67` allocates a destination with `MLMultiArray(shape:dataType:)`, derives `rowBytes` from the source byte count, then copies all retained rows together.
- WhisperKit `Sources/WhisperKit/Core/TextDecoder.swift:141` allocates its float16 alignment buffer using the `initialValue:` initializer. `Sources/ArgmaxCore/MLMultiArrayExtensions.swift:11-18` implements that float16 initializer with an IOSurface-backed pixel buffer.
- `TextDecoder.swift:784` and `TranscribeTask.swift:198-203` pass this buffer to the segment seeker.
Release-mode reproduction using the exact upstream initializer and unchanged production row-shift code:
| buffer, shape 224 × 1500 | strides | exposed bytes |
|---|---|---:|
| upstream float16 source | `[1504, 1]` | 673792 |
| ordinary destination | `[1500, 1]` | 672000 |
After filling each source row's first element with its row index and shifting by four rows, destination `[219, 0]` is **0**, where the expected source `[223, 0]` is **223**.
A second reproduction uses the existing synthetic alignment fixture and the real WhisperKit `SegmentSeeker`, changing the prompted buffer allocation to the upstream IOSurface initializer and populating it by logical indices. The first word's expected interval is **0.12–0.16 s**; the prompted result is **0.72–0.96 s**. All five word intervals differ. The equivalent existing test with ordinary dense allocation passes.
## Fix and acceptance criteria
Copy logical alignment elements or rows using the actual source and destination strides. Preserve shape and data type, and zero the trailing logical rows. Do not copy padding as data or assume equal physical row sizes merely because shapes match.
Add a regression using the upstream float16 initializer, including a width with padded rows such as 1500. Verify every retained logical row and the zeroed tail. Extend the real-seeker equivalence test to cover this allocation, retaining the ordinary dense-buffer case. Validate a prompted audio decode after the fix to assess its effect on segment retention and seek positions.
Related: closed #423 introduced prompt-offset alignment. This is a distinct defect in the resulting row-copy implementation: the offset is applied, but unequal physical strides corrupt the shifted matrix. Existing tests allocate dense buffers and miss this boundary.
Priority: P2. Two release-mode reproductions fail their correctness assertions; production source is unchanged.
Contributor guide
Research direction
Start in Sources/UttrflowSpeech/PromptAlignedSegmentSeeker.swift:54-67 and compare its row copy with the source and destination strides. Run the existing synthetic alignment and real-seeker equivalence tests, then add coverage using WhisperKit’s float16 initializer with a padded width such as 1500. Done means retained logical rows match and the tail is zeroed without copying padding, with the prompted decode behavior checked afterward.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- macos, swift
- Domain
- audio-video-rtc, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100