huggingface / huggingface/swift-transformers

CoreML Cancellation Crash In `swift-transformers`

Open
#364 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
1.4k
Forks
209
Avg merge
2d 7h
Merged PRs (30d)
4

Description

## Summary

When using `swift-transformers` with a local Core ML language model, cancelling an in-flight generation request can crash the entire process instead of surfacing a recoverable Swift error.

The immediate cause appears to be `try! await model.prediction(from:)` inside `LanguageModel.predictNextTokenScores(...)`. If Core ML throws `CancellationError`, the `try!` triggers a fatal error.

## Environment & System Under Test (SUT)

- **Device/Hardware**: Apple Silicon Mac (M-series) / iPad
- **OS Version**: macOS Tahoe 26.5 (Build 25F71) / iOS 18+
- **Xcode Version**: Xcode 26.5 (Build 17F42)
- **Target SDKs**: macOS 14.4+ / iOS 17.0+
- **Swift Compiler**: Swift 6.0 (Strict Concurrency: Complete)
- **Model Runtime**: local compiled Core ML model loaded through `Models.LanguageModel`
- **App Integration**: Swift client application wrapping compiled Core ML models directly
- **Model Characteristics**:
- causal LM
- rank-3 logits
- local Core ML model, not remote inference
- model ABI validated successfully before generation

## Runtime Dependency & Tooling Versions

- **`swift-transformers`**: 1.1.6 (huggingface package revision: `573e5c9`)
- **Hugging Face `AnyLanguageModel`**: 0.8.0 (revision: `163f385`)
- **Python Environment**: Python 3.10.16
- **`coremltools` (Converter)**: 7.1 (compiled with PyTorch 2.2.1)

## Development Dependency & Tooling Versions for model conversion
- **`python`**: 3.11.15
- **`coremltools`** : 9.0
- **`transformers`** : 5.8.1
- **`tokenizers`** : 0.22.2
- **`torch`** : 2.7.0
-
## Observed Behavior

Generation begins normally, then the process crashes with:

- `Fatal error: 'try!' expression unexpectedly raised an error: Swift.CancellationError()`

Relevant upstream line:

- `Sources/Models/LanguageModel.swift`
- `let outputs = try! await model.prediction(from: inputDictionary)`

Representative stack shape:

- `LanguageModel.predictNextTokenScores(_:config:)`
- `Generation.generate(...)`
- Client application's structured `Task` executing generation

The crash happens after generation has already started, not during model load or ABI validation.

## Expected Behavior

If a caller cancels an in-flight Core ML generation request, one of the following should happen:

1. cancellation propagates as a normal thrown Swift error, or
2. cancellation is explicitly handled by the library in a documented way

It should not fatal-crash the process.

## Why This Matters

This makes local Core ML generation unsafe to wrap in normal Swift concurrency patterns such as:

- timeout races
- task cancellation on view/session teardown
- queue-level cancellation during backlog recovery

In practice, callers cannot safely apply structured cancellation around local Core ML generation, because a routine cancellation may terminate the app.

## Suspected Trigger

The issue appears to be specifically cancellation-safety in the Core ML prediction path, not model compatibility:

- the model validates successfully
- generation starts successfully
- a `CancellationError` is thrown during prediction
- `try!` converts that into a fatal crash

This suggests the immediate bug is not “Core ML should never throw cancellation,” but rather that `swift-transformers` should not use `try!` in this path.

## Minimal Behavioral Difference

Stable:

- allow local Core ML generation to run to completion
- avoid cancelling the in-flight generation task

Crashes:

- cancel the generation task while `model.prediction(from:)` is still in flight

## Representative Client Application Scenario

One reproducible trigger is racing generation inside a structured Swift concurrency Task against a timeout:

1. Start an asynchronous task executing local Core ML model generation via `Generation.generate(...)`
2. Race it against a timeout/sleep Task
3. If the timeout task wins, cancel the generation Task
4. Under the hood, Core ML throws `CancellationError`
5. `swift-transformers` crashes the process in `predictNextTokenScores` due to `try!`

This is a normal and reasonable use of Swift structured concurrency from the caller side.

## Crash Signature

Representative fatal:

```text
Fatal error: 'try!' expression unexpectedly raised an error: Swift.CancellationError()
```

Representative stack shape:

```text
LanguageModel.predictNextTokenScores(_:config:) at LanguageModel.swift:75
Generation.generate(...)
ClientApplication.runGenerationTask(...)
```

## Suggested Direction

The most direct fix would be to replace the `try!` in the Core ML prediction path with normal error propagation or explicit cancellation handling.

Even if the library chooses not to guarantee graceful mid-generation cancellation semantics, callers should still receive a recoverable error instead of a fatal crash.

## Workaround

We are currently avoiding timeout-driven cancellation entirely for local Core ML models. That prevents the crash, but it also means callers must give up normal cancellation behavior for this backend.

## Why This Is High Impact

This affects a normal async control-flow pattern and can crash an otherwise healthy app even when:

- the model is valid
- generation has started correctly
- the caller is only trying to cancel work responsibly

That makes the issue easy for downstream apps to hit once they add timeouts, queue cancellation, or teardown cleanup around local Core ML generation.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.