apple / apple/coreai-models

[Swift runtime] Pipelined prefix reuse skips the last sampled token

Closed
#234 1 comment 0 reactions 1 assignee Claimed by @stikves View on GitHub
Dominant language
Swift
Stars
2.1k
Forks
202
Avg merge
2d 8h
Merged PRs (30d)
55

Description

### Model name

Qwen3.5 0.8B GDN S=1, Qwen3.8 27B GDN S=1, and dynamic Qwen3 0.6B (public bundle revisions and full file hashes in the linked evidence).

### What happens

On official `27a66f90e7f3fd9b83a6acb7bcb0a4a5ff71fd60`, a completed 24-token generation from a 24-token prompt leaves `processedTokenCount == 47`, while history contains 48 tokens. Extending that history to 68 input tokens reports a prefix hit of 48 and skips the last sampled token. After another 24 outputs, the processed count is 90; a full replay gives the expected 91.

The 27B greedy continuation differs from full replay (“The Eiffel Tower…” vs “One famous landmark in Paris…”). The count discrepancy also occurs on the dynamic control before #227, so I am reporting this separately from #212/#227. Full reset/replay is repeatable.

A four-line patch limits reusable history to `processedTokenCount`. The same three fixtures then pass count and token parity; all seven dynamic generation phases remain token-identical. Hybrid partial reset remains explicitly rejected. [Review copy of patch and opt-in regression test](https://github.com/john-rocky/coreai-models/pull/1). Related test-coverage tracking: #88. This is not a claim covering cancellation/early-EOS or every multi-turn case.

### Command run

The following creates an isolated reproduction checkout and reverses only the patch, retaining the consumer test package. Run with a platform-compatible macOS bundle and your shared GPU lock held:

```sh
REPRO_DIR=$(mktemp -d)
git clone --branch fix/pipelined-pending-token https://github.com/john-rocky/coreai-models.git "$REPRO_DIR"
cd "$REPRO_DIR"
git checkout --detach a40d26eaeb7dd5492c24edc178929e8e40daa184
git revert --no-commit 48281854f9834cdcc61b9cb320a960835eb1e8f2
export DEVELOPER_DIR=/Applications/Xcode-27.0.0-Beta.5.app
export COREAI_CONTINUATION_BUNDLE=/absolute/path/to/macos/bundle
unset COREAI_CHUNK_THRESHOLD
swift test --package-path validation/2026-09-05-coreai/consumer -c release --force-resolved-versions --filter PipelinedContinuationTests
```

The regression covers budgets 1 and 4, successive extensions, exact continuation and full replay. It fails on the unpatched source and passes with the patch on the 0.8B S=1 and dynamic fixtures. Separate logs include 24-token text generation on the 27B fixture.

### macOS / iOS target

Mac Studio `Mac16,9`, Apple M4 Max, 128 GiB; macOS 27.0 `26A5416b`. No iPhone run.

### Xcode version

Xcode 27 beta 5 `27A5237l`; macOS SDK 27.0 `26A5406c`; Swift 6.4.

### Full error output and evidence

[Results, exact revisions, commands and downloadable logs](https://github.com/john-rocky/coreai-models/blob/a40d26eaeb7dd5492c24edc178929e8e40daa184/validation/2026-09-05-coreai/RESULTS.md). The archive is 109,859 bytes with SHA-256 `a63d2700eed17c0e0b7c6b9242b40249cce792afbc3856e1b5df2b1b2a7a1913`.

The new regression test records `processedTokenCount` and prefix-hit expectation failures on the unpatched engine. For example, the third turn processes 16 tokens where 18 are expected, and the 0.8B output also diverges from replay in that case. The corrected engine passes both cases.

I am using an issue with a forked review copy in accordance with the repository's current contribution policy.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.