huggingface / huggingface/swift-transformers
CoreML Cancellation Crash In `swift-transformers`
- Dominant language
- Swift
- Stars
- 1.4k
- Forks
- 209
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 4
Description
## Summary
When using `swift-transformers` with a local Core ML language model, cancelling an in-flight generation request can crash the entire process instead of surfacing a recoverable Swift error.
The immediate cause appears to be `try! await model.prediction(from:)` inside `LanguageModel.predictNextTokenScores(...)`. If Core ML throws `CancellationError`, the `try!` triggers a fatal error.
## Environment & System Under Test (SUT)
- **Device/Hardware**: Apple Silicon Mac (M-series) / iPad
- **OS Version**: macOS Tahoe 26.5 (Build 25F71) / iOS 18+
- **Xcode Version**: Xcode 26.5 (Build 17F42)
- **Target SDKs**: macOS 14.4+ / iOS 17.0+
- **Swift Compiler**: Swift 6.0 (Strict Concurrency: Complete)
- **Model Runtime**: local compiled Core ML model loaded through `Models.LanguageModel`
- **App Integration**: Swift client application wrapping compiled Core ML models directly
- **Model Characteristics**:
- causal LM
- rank-3 logits
- local Core ML model, not remote inference
- model ABI validated successfully before generation
## Runtime Dependency & Tooling Versions
- **`swift-transformers`**: 1.1.6 (huggingface package revision: `573e5c9`)
- **Hugging Face `AnyLanguageModel`**: 0.8.0 (revision: `163f385`)
- **Python Environment**: Python 3.10.16
- **`coremltools` (Converter)**: 7.1 (compiled with PyTorch 2.2.1)
## Development Dependency & Tooling Versions for model conversion
- **`python`**: 3.11.15
- **`coremltools`** : 9.0
- **`transformers`** : 5.8.1
- **`tokenizers`** : 0.22.2
- **`torch`** : 2.7.0
-
## Observed Behavior
Generation begins normally, then the process crashes with:
- `Fatal error: 'try!' expression unexpectedly raised an error: Swift.CancellationError()`
Relevant upstream line:
- `Sources/Models/LanguageModel.swift`
- `let outputs = try! await model.prediction(from: inputDictionary)`
Representative stack shape:
- `LanguageModel.predictNextTokenScores(_:config:)`
- `Generation.generate(...)`
- Client application's structured `Task` executing generation
The crash happens after generation has already started, not during model load or ABI validation.
## Expected Behavior
If a caller cancels an in-flight Core ML generation request, one of the following should happen:
1. cancellation propagates as a normal thrown Swift error, or
2. cancellation is explicitly handled by the library in a documented way
It should not fatal-crash the process.
## Why This Matters
This makes local Core ML generation unsafe to wrap in normal Swift concurrency patterns such as:
- timeout races
- task cancellation on view/session teardown
- queue-level cancellation during backlog recovery
In practice, callers cannot safely apply structured cancellation around local Core ML generation, because a routine cancellation may terminate the app.
## Suspected Trigger
The issue appears to be specifically cancellation-safety in the Core ML prediction path, not model compatibility:
- the model validates successfully
- generation starts successfully
- a `CancellationError` is thrown during prediction
- `try!` converts that into a fatal crash
This suggests the immediate bug is not “Core ML should never throw cancellation,” but rather that `swift-transformers` should not use `try!` in this path.
## Minimal Behavioral Difference
Stable:
- allow local Core ML generation to run to completion
- avoid cancelling the in-flight generation task
Crashes:
- cancel the generation task while `model.prediction(from:)` is still in flight
## Representative Client Application Scenario
One reproducible trigger is racing generation inside a structured Swift concurrency Task against a timeout:
1. Start an asynchronous task executing local Core ML model generation via `Generation.generate(...)`
2. Race it against a timeout/sleep Task
3. If the timeout task wins, cancel the generation Task
4. Under the hood, Core ML throws `CancellationError`
5. `swift-transformers` crashes the process in `predictNextTokenScores` due to `try!`
This is a normal and reasonable use of Swift structured concurrency from the caller side.
## Crash Signature
Representative fatal:
```text
Fatal error: 'try!' expression unexpectedly raised an error: Swift.CancellationError()
```
Representative stack shape:
```text
LanguageModel.predictNextTokenScores(_:config:) at LanguageModel.swift:75
Generation.generate(...)
ClientApplication.runGenerationTask(...)
```
## Suggested Direction
The most direct fix would be to replace the `try!` in the Core ML prediction path with normal error propagation or explicit cancellation handling.
Even if the library chooses not to guarantee graceful mid-generation cancellation semantics, callers should still receive a recoverable error instead of a fatal crash.
## Workaround
We are currently avoiding timeout-driven cancellation entirely for local Core ML models. That prevents the crash, but it also means callers must give up normal cancellation behavior for this backend.
## Why This Is High Impact
This affects a normal async control-flow pattern and can crash an otherwise healthy app even when:
- the model is valid
- generation has started correctly
- the caller is only trying to cancel work responsibly
That makes the issue easy for downstream apps to hit once they add timeouts, queue cancellation, or teardown cleanup around local Core ML generation.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.