feat(runtime): materialize native PDF inputs for verified PDF-capable providers
- Dominant language
- TypeScript
- Stars
- 5.4k
- Forks
- 502
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 715
Description
## Problem
Maka synchronizes PDF-input metadata and exposes `resolveModelPdfSupport`, but the runtime never consumes that capability. `AiSdkBackend.appendImageParts` filters attachments to `kind === image`; a PDF remains only a text placeholder even when the selected provider/model has a verified native PDF contract.
The storage side is already present: PDF attachments have a canonical kind and MIME type, are persisted as Session artifacts, and can be read through the existing bounded `AttachmentByteReader`. The missing boundary is provider request materialization.
## Provider research
AI SDK encoding support and provider support are separate facts. The current locked SDK produces these request shapes from one generic `file` part:
- OpenAI-compatible Chat: `type: file` with `filename` and Base64 `file_data`;
- Anthropic Messages: `type: document` with a Base64 `application/pdf` source;
- OpenAI Responses: `type: input_file` with `filename` and Base64 `file_data`.
That proves the SDK can lower a generic file part. It does not prove every OpenAI-compatible provider accepts the resulting wire.
### Verified first-party contracts
- **Anthropic Messages:** all active Claude models support PDF processing. The API accepts URL, Base64, or Files API references. The complete request is limited to 32 MB; PDFs must be standard and unencrypted; the page limit is 600, or 100 when the request context window is below 1M tokens.
- **OpenAI:** native Chat Completions and Responses accept PDF file inputs. Responses uses `input_file`; PDF processing includes extracted text and page images. Each file and the combined files in one request must remain below 50 MB.
### Not yet verified
- **Generic OpenAI-compatible providers:** the adapter can emit the OpenAI Chat `file_data` shape, but each provider must opt in only after its own contract is verified. Unknown compatible providers must remain unsupported.
- **Alibaba Token Plan / `qwen3.8-max`:** Alibaba Chat documentation lists text, image, audio, and video content but no `file`/`file_data` PDF shape. Alibaba Responses documents `input_file`, but currently states that this content type is supported only by `qwen3.5-ocr` and requires `file_url`. Therefore `qwen3.8-max` is not an acceptance target for this issue. Its generated `pdf` modality is insufficient authority for sending local PDF bytes.
- Alibaba PDF support, if added later, needs a separate provider contract. If the service continues to require `file_url`, it also needs a safe upload/temporary-URL design rather than silently exposing local files.
## Scope status
This is an independent Runtime attachment-capability issue. It is **not** a follow-up requirement for Qwen3.8 support, #3156, #3157, #3162, or #3163, and it does not block those deliverables. Alibaba/Qwen remains in this issue only as a negative boundary showing why generic OpenAI-compatible PDF support must not be inferred without provider evidence.
## Desired outcome
A model on a **verified PDF-capable provider/wire** receives an attached PDF as a provider-native file part on the current Turn and during durable replay. Unsupported, unknown, or unverified provider/model combinations continue to receive the bounded placeholder and never receive raw PDF bytes.
## Proposed delivery
### First slice: Anthropic and native OpenAI
1. Replace the unused boolean-only assumption with an explicit provider/wire PDF-input contract. Do not infer generic-compatible support from a model name or generated modality alone.
2. Carry the resolved contract into `AiSdkBackend` from the production Runtime Host composition. The former Headless production path no longer exists on current `main` and is not part of this slice.
3. Materialize `AttachmentRef.kind === pdf` through the existing attachment byte reader as one generic AI SDK `file` part with exact `application/pdf`, filename, and bytes. Let the verified AI SDK provider adapter perform the wire-specific lowering.
4. Cover native Anthropic Messages and native OpenAI Chat/Responses. Keep all generic OpenAI-compatible providers disabled unless separately verified.
5. Keep current Turn, RuntimeEvent replay, StoredMessage fallback, steering, compaction replacement, and restart behavior consistent.
### Budget and failure policy
- Add a PDF sub-budget and a combined raw-binary attachment budget. Separate independent image/PDF limits are not sufficient because Anthropic applies its 32 MB cap to the complete request and Base64 adds roughly one-third overhead.
- Preserve the existing 12 MB image sub-limit. Choose the PDF and combined limits from measured encoded request size, leaving room for text, schemas, JSON, and other attachments.
- Deduplicate budget decisions by durable occurrence across provider steps, as the image path does today.
- Missing, unreadable, oversized, unsupported, or locally invalid PDFs must degrade to a safe bounded explanation; PDF bytes must never enter transcript text, logs, RuntimeEvents, or diagnostics.
- Preflight parsing for encrypted/page-limit PDFs is tracked in #3284; bounded recovery after provider rejection is tracked in #3285. Neither is a merge gate for this first slice. Ordinary provider errors remain governed by the existing bounded `ModelFailure` handling.
## Acceptance criteria
- `resolveModelPdfSupport` is no longer dead metadata; the runtime has an explicit verified provider/wire gate.
- Native Anthropic receives a `document` block with Base64 `application/pdf` data through the SDK.
- Native OpenAI Chat receives `type: file`; native OpenAI Responses receives `type: input_file`.
- A generic OpenAI-compatible connection does not receive PDF bytes by default.
- Current Turn, RuntimeEvent replay, StoredMessage fallback, steering, compaction, and restart all present the same durable PDF occurrence without duplicating the user message.
- Image and PDF attachments respect both their subtype caps and the combined binary-request budget.
- Unsupported models and local attachment failures keep the safe placeholder. Provider request failures use the existing bounded `ModelFailure` path; this slice does not add PDF-specific automatic retry.
- Focused live verification covers at least one native Anthropic model and one native OpenAI model when protected credentials are available.
## Non-goals
- Claiming PDF support for `qwen3.8-max` or another generic-compatible model without provider evidence.
- Uploading local PDFs to a third-party file service or creating public temporary URLs.
- OCR fallback for models without native PDF support.
- Automatically extracting PDF text into the prompt.
- Treating arbitrary Office or binary files as provider-safe PDF/document inputs.
- Preflight parsing for encrypted PDFs/page counts (#3284) or automatic PDF-specific retry after provider rejection (#3285).
## References
- OpenAI file inputs: https://developers.openai.com/api/docs/guides/file-inputs
- Anthropic PDF support: https://platform.claude.com/docs/en/build-with-claude/pdf-support
- Alibaba Chat API: https://help.aliyun.com/zh/model-studio/qwen-api-via-openai-chat-completions
- Alibaba Responses API: https://help.aliyun.com/zh/model-studio/qwen-api-via-openai-responses
Disclosure: this issue was drafted and revised with OpenAI Codex assistance and reviewed and approved by the human contributor.
Contributor guide
Assessment
This issue has not been assessed yet.