apple / apple/coreai-models

Guided generation refused for every chunked-static (iOS) export after the first request

Closed
#247 1 comment 0 reactions 1 assignee Claimed by @stikves View on GitHub
Dominant language
Swift
Stars
2.1k
Forks
202
Avg merge
2d 8h
Merged PRs (30d)
55

Description

## What happens

Guided generation is refused for chunked-static exports, which is every iOS
export — but only *after* the first request of a session.

Device: iPhone 15 Pro Max, iOS 27.0. Artifact: `Qwen2.5-Coder-1.5B-Instruct`
exported with `coreai.llm.export ... --platform iOS --experimental
--compute-precision float16`. Twelve structured requests through
`LanguageModelSession.respond(to:generating:)`: the first returned a correct
answer, the remaining eleven failed with

```
The selected model does not support guided generation. Consider trying again
with a different model.
```

An earlier artifact (`Qwen3-1.7B`, the `qwen3_1_7b_6bit` preset) showed the same
thing in a different shape: `GeneratedContent does not contain a property
'isCodingRequest'. Content: {}`.

Plain, unstructured generation works on both artifacts and is coherent.

## Cause

`CoreAILanguageModel.isGuidedGenerationSupported` asks whether the resident
engine conforms to `ConstrainedGenerationCapable`, and returns that answer:

```swift
private var isGuidedGenerationSupported: Bool {
if let isConstrainedCapable = resources.loadedEngineIsConstrainedCapable {
return isConstrainedCapable
}
if let supportsLogits = resources.loadedEngineSupportsLogits {
return supportsLogits
}
return true
}
```

`StaticShapeEngine` does not conform to `ConstrainedGenerationCapable`, and does
report `supportsLogits == true`, so the second check is never reached. The
executor already treats the two as alternatives:

```swift
guard engine.supportsLogits || engine is any ConstrainedGenerationCapable else {
throw LanguageModelError.unsupportedCapability(.init(capability: .guidedGeneration, ...))
}
```

The delayed onset is the third branch: with no engine resident both properties
are nil and the model answers `true`, so the first request proceeds and
succeeds.

## Suggested change

Accept either capability, keeping "assume yes" only while nothing is loaded:

```swift
private var isGuidedGenerationSupported: Bool {
let constrainedCapable = resources.loadedEngineIsConstrainedCapable
let supportsLogits = resources.loadedEngineSupportsLogits
if constrainedCapable == nil && supportsLogits == nil { return true }
return (constrainedCapable ?? false) || (supportsLogits ?? false)
}
```

With that change, the same twelve cases return twelve structured answers on the
same artifact and device. Pull requests appear to be closed to outside
contributors, so this is an issue rather than a patch; the change is running
here at mohdelfatih/coreai-models@66531257de45f3c1ffa7618e2950242b853c062d if it
is useful.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.