apache / apache/maka

feat(runtime): materialize native PDF inputs for verified PDF-capable providers

Open
#3,164 5 comments 0 reactions 1 assignee Claimed by @tianrking View on GitHub
enhancement
Dominant language
TypeScript
Stars
5.4k
Forks
502
Avg merge
1d 2h
Merged PRs (30d)
715

Description

## Problem

Maka synchronizes PDF-input metadata and exposes `resolveModelPdfSupport`, but the runtime never consumes that capability. `AiSdkBackend.appendImageParts` filters attachments to `kind === image`; a PDF remains only a text placeholder even when the selected provider/model has a verified native PDF contract.

The storage side is already present: PDF attachments have a canonical kind and MIME type, are persisted as Session artifacts, and can be read through the existing bounded `AttachmentByteReader`. The missing boundary is provider request materialization.

## Provider research

AI SDK encoding support and provider support are separate facts. The current locked SDK produces these request shapes from one generic `file` part:

- OpenAI-compatible Chat: `type: file` with `filename` and Base64 `file_data`;
- Anthropic Messages: `type: document` with a Base64 `application/pdf` source;
- OpenAI Responses: `type: input_file` with `filename` and Base64 `file_data`.

That proves the SDK can lower a generic file part. It does not prove every OpenAI-compatible provider accepts the resulting wire.

### Verified first-party contracts

- **Anthropic Messages:** all active Claude models support PDF processing. The API accepts URL, Base64, or Files API references. The complete request is limited to 32 MB; PDFs must be standard and unencrypted; the page limit is 600, or 100 when the request context window is below 1M tokens.
- **OpenAI:** native Chat Completions and Responses accept PDF file inputs. Responses uses `input_file`; PDF processing includes extracted text and page images. Each file and the combined files in one request must remain below 50 MB.

### Not yet verified

- **Generic OpenAI-compatible providers:** the adapter can emit the OpenAI Chat `file_data` shape, but each provider must opt in only after its own contract is verified. Unknown compatible providers must remain unsupported.
- **Alibaba Token Plan / `qwen3.8-max`:** Alibaba Chat documentation lists text, image, audio, and video content but no `file`/`file_data` PDF shape. Alibaba Responses documents `input_file`, but currently states that this content type is supported only by `qwen3.5-ocr` and requires `file_url`. Therefore `qwen3.8-max` is not an acceptance target for this issue. Its generated `pdf` modality is insufficient authority for sending local PDF bytes.
- Alibaba PDF support, if added later, needs a separate provider contract. If the service continues to require `file_url`, it also needs a safe upload/temporary-URL design rather than silently exposing local files.

## Scope status

This is an independent Runtime attachment-capability issue. It is **not** a follow-up requirement for Qwen3.8 support, #3156, #3157, #3162, or #3163, and it does not block those deliverables. Alibaba/Qwen remains in this issue only as a negative boundary showing why generic OpenAI-compatible PDF support must not be inferred without provider evidence.

## Desired outcome

A model on a **verified PDF-capable provider/wire** receives an attached PDF as a provider-native file part on the current Turn and during durable replay. Unsupported, unknown, or unverified provider/model combinations continue to receive the bounded placeholder and never receive raw PDF bytes.

## Proposed delivery

### First slice: Anthropic and native OpenAI

1. Replace the unused boolean-only assumption with an explicit provider/wire PDF-input contract. Do not infer generic-compatible support from a model name or generated modality alone.
2. Carry the resolved contract into `AiSdkBackend` from the production Runtime Host composition. The former Headless production path no longer exists on current `main` and is not part of this slice.
3. Materialize `AttachmentRef.kind === pdf` through the existing attachment byte reader as one generic AI SDK `file` part with exact `application/pdf`, filename, and bytes. Let the verified AI SDK provider adapter perform the wire-specific lowering.
4. Cover native Anthropic Messages and native OpenAI Chat/Responses. Keep all generic OpenAI-compatible providers disabled unless separately verified.
5. Keep current Turn, RuntimeEvent replay, StoredMessage fallback, steering, compaction replacement, and restart behavior consistent.

### Budget and failure policy

- Add a PDF sub-budget and a combined raw-binary attachment budget. Separate independent image/PDF limits are not sufficient because Anthropic applies its 32 MB cap to the complete request and Base64 adds roughly one-third overhead.
- Preserve the existing 12 MB image sub-limit. Choose the PDF and combined limits from measured encoded request size, leaving room for text, schemas, JSON, and other attachments.
- Deduplicate budget decisions by durable occurrence across provider steps, as the image path does today.
- Missing, unreadable, oversized, unsupported, or locally invalid PDFs must degrade to a safe bounded explanation; PDF bytes must never enter transcript text, logs, RuntimeEvents, or diagnostics.
- Preflight parsing for encrypted/page-limit PDFs is tracked in #3284; bounded recovery after provider rejection is tracked in #3285. Neither is a merge gate for this first slice. Ordinary provider errors remain governed by the existing bounded `ModelFailure` handling.

## Acceptance criteria

- `resolveModelPdfSupport` is no longer dead metadata; the runtime has an explicit verified provider/wire gate.
- Native Anthropic receives a `document` block with Base64 `application/pdf` data through the SDK.
- Native OpenAI Chat receives `type: file`; native OpenAI Responses receives `type: input_file`.
- A generic OpenAI-compatible connection does not receive PDF bytes by default.
- Current Turn, RuntimeEvent replay, StoredMessage fallback, steering, compaction, and restart all present the same durable PDF occurrence without duplicating the user message.
- Image and PDF attachments respect both their subtype caps and the combined binary-request budget.
- Unsupported models and local attachment failures keep the safe placeholder. Provider request failures use the existing bounded `ModelFailure` path; this slice does not add PDF-specific automatic retry.
- Focused live verification covers at least one native Anthropic model and one native OpenAI model when protected credentials are available.

## Non-goals

- Claiming PDF support for `qwen3.8-max` or another generic-compatible model without provider evidence.
- Uploading local PDFs to a third-party file service or creating public temporary URLs.
- OCR fallback for models without native PDF support.
- Automatically extracting PDF text into the prompt.
- Treating arbitrary Office or binary files as provider-safe PDF/document inputs.
- Preflight parsing for encrypted PDFs/page counts (#3284) or automatic PDF-specific retry after provider rejection (#3285).

## References

- OpenAI file inputs: https://developers.openai.com/api/docs/guides/file-inputs
- Anthropic PDF support: https://platform.claude.com/docs/en/build-with-claude/pdf-support
- Alibaba Chat API: https://help.aliyun.com/zh/model-studio/qwen-api-via-openai-chat-completions
- Alibaba Responses API: https://help.aliyun.com/zh/model-studio/qwen-api-via-openai-responses

Disclosure: this issue was drafted and revised with OpenAI Codex assistance and reviewed and approved by the human contributor.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.