Structured-output grammar compilation causes 30-45s first-token stalls; cache misses make it recurrent — document and mitigate

## Summary

Guided generation through this bridge maps `GenerationSchema` onto the Messages API's structured outputs (`output_config.format`). That server-side constrained decoding compiles the JSON Schema into a grammar before the first token can stream, and in practice this dominates the latency profile of any interactive FoundationModels app that switches from the on-device model to Claude. It deserves a mention in the README, and ideally a mitigation hook, because the symptom looks exactly like "Claude is slow" when neither the model nor the network is at fault.

## What we measured

App context: a live language-lookup app firing a `respond(generating:)` request per typing pause, `@Generable` payload of ~5.4KB encoded schema (nested `$defs`, arrays of structs), `claude-sonnet-4-6`, adaptive thinking, `effort: low`, streaming.

**Cold schema compile.** First request with a new schema structure: 33-48s elapsed before `message_start`, with trivial generation afterward (~300 output tokens). Reproduced repeatedly. Editing `@Guide` description text does not re-trigger the compile (the cache keys on structure), but any structural change (add/remove/rename a property) re-pays it in full. During iterative development on `@Generable` types this means the first request after every build feels catastrophically broken.

**The cache is not reliable even for untouched schemas.** The API docs say compiled grammars are cached for 24 hours from last use, but we observed a mid-session 27.5s time-to-first-byte on a schema that had served fast requests minutes earlier (`request-id` available on request). A controlled probe (66 requests over 45 minutes, alternating identical requests with and without `output_config.format`, same model/system/tools/prompt cadence) showed:

| Request type | median TTFB | max TTFB | stalls >8s |
|---|---|---|---|
| no schema | 1.17s | 2.67s | 0 of 22 |
| with schema | 3.29s | 9.28s | 3 of 44 |

Stall request-ids: `req_011Cd1zkmq34MBWkkoY7w5gS`, `req_011Cd21U2tnuorgopPodrVVi`, `req_011Cd21qRNEM7h3E7qgnvvBL` (2026-07-14, 21:40-22:25 PT).

So structured outputs cost roughly +2s median on every request versus schema-free, plus a fat tail of multi-second to multi-ten-second pre-stream stalls that are uncorrelated with idle time or prompt content.

## Why this package is the right place to address it

FoundationModels developers adopt guided generation as the default (it is the framework's flagship feature), and on the system model constrained decoding is effectively free. When they swap in `ClaudeLanguageModel`, every `respond(generating:)` silently inherits the grammar-compile tax, and nothing in the API surface hints at where the time went. The failure mode is a developer concluding the model is slow and abandoning the bridge.

## Suggestions

1. **Document it.** A README section on structured-output latency: one-time compile cost per schema structure, what busts the cache (structure, not descriptions), and the observed stall tail.
2. **A prewarm affordance.** `LanguageModelSession.prewarm()` exists for the on-device model; a bridge-level analog could fire a minimal structured request to compile the grammar off the critical path at app launch. Note `max_tokens: 0` (the cache-prewarm idiom) is rejected in combination with `output_config.format`, so this needs a real micro-request.
3. **Diagnosability.** Logging time-to-first-event and the `request-id` response header (we patched this locally) turns "it feels slow" into an actionable report. Consider shipping something like it behind a debug flag.
4. **Optionally, a first-byte watchdog.** Since retries overwhelmingly land warm (~2s), an opt-in "no first byte within N seconds, cancel and re-fire once" policy would convert the stall tail into a bounded worst case.

Happy to share the raw probe CSV, the exact sanitized schema, and additional request-ids.