Add OpenAI-compatible computer use tool support

Problem statement / motivation

The Server-Side Tool Execution roadmap lists Computer use as an expected tool area, but agentic-api does not currently have first-class support for the OpenAI Responses API computer tool.

The Rust request model has no typed computer tool variant, and the execution path has no representation for computer_call, computer_call_output, screenshots, or computer actions. As a result, clients cannot use the current OpenAI-compatible computer-use loop through agentic-api, and deployments cannot register a gateway-executed browser or VM harness.

OpenAI's current computer-use flow is:

1. Send a Responses request with tools: [{"type": "computer"}].
2. Receive a computer_call containing an ordered actions[] batch.
3. Execute those actions in a browser or VM.
4. Capture the updated screen and submit it as computer_call_output.
5. Continue until the model no longer emits a computer_call.

This fits the roadmap's tool loop and ownership model. A configured computer harness can be a gateway-executed built-in tool; without a configured handler, the gateway must not execute UI actions by default.

Proposed solution

Add first-class, OpenAI-compatible support for the current computer built-in tool.

Wire types

• Accept and preserve the request declaration {"type": "computer"}.
• Add typed input/output items for:
  • computer_call
  • computer_call_output
  • computer_screenshot
• Represent the current action set:
  • click
  • double_click
  • scroll
  • type
  • wait
  • keypress
  • drag
  • move
  • screenshot
• Keep the action representation non-exhaustive so future action types can be preserved safely.
• Preserve call IDs, action ordering, statuses, screenshots, and image detail across storage and continuation.

Tool routing and execution

• Register computer in the request-scoped tool registry.
• If upstream inference requires function-shaped tools, normalize the declaration for inference and restore model calls to the public typed computer_call shape.
• Execute computer calls only when an explicit gateway computer handler is configured.
• Run every action in actions[] in order, capture a full screenshot after the batch, append computer_call_output, and continue the inference loop.
• When no gateway handler is configured, preserve/return the call for client execution or pass it through according to the existing ownership rules. Never control the gateway host UI by default.
• Support both blocking and streaming Responses flows, including continuation through previous_response_id and stored response state.

Safety and configuration

• Require an isolated browser, container, or VM for gateway execution.
• Do not inherit host environment variables or expose the host filesystem by default.
• Support URL/domain allowlists, timeouts, action/round limits, cancellation, and audit logging.
• Treat screenshots, pages, documents, emails, chats, and other third-party content as untrusted input.
• Provide a human-review/approval hook for high-impact actions and sensitive-data transmission.

Acceptance criteria

• {"type": "computer"} round-trips through request parsing, storage, rehydration, and serialization.
• All documented computer action variants deserialize and serialize without losing fields or order.
• computer_call and computer_call_output are supported in blocking and streaming Responses flows.
• A configured test harness can complete a multi-round action/screenshot loop and produce a final assistant response.
• previous_response_id continuation preserves the computer call and its output.
• A request without a configured gateway handler never executes UI actions implicitly.
• Timeouts, cancellation, malformed actions, handler failures, and maximum-loop exhaustion return typed, test-covered failures.
• Documentation explains handler registration, isolation, approvals, and the client-executed fallback.
• Compatibility tests cover the OpenAI request and output shapes below.

Alternatives considered

No response

Additional context

OpenAI API examples

The model name below follows the current OpenAI documentation; a vLLM deployment would use a model capable of emitting the compatible tool-call shape.

Start a computer-use request

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1")

response = client.responses.create(
    model="gpt-5.6",
    tools=[{"type": "computer"}],
    input=(
        "Check whether the Filters panel is open. If it is not, click Show filters. "
        "Then type penguin in the search box. Use the computer tool for UI interaction."
    ),
)

The first output may request a screenshot:

{
  "type": "computer_call",
  "call_id": "call_001",
  "actions": [
    { "type": "screenshot" }
  ],
  "status": "completed"
}

Return the updated screenshot

response = client.responses.create(
    model="gpt-5.6",
    tools=[{"type": "computer"}],
    previous_response_id=response.id,
    input=[
        {
            "type": "computer_call_output",
            "call_id": "call_001",
            "output": {
                "type": "computer_screenshot",
                "image_url": f"data:image/png;base64,{screenshot_base64}",
                "detail": "original",
            },
        }
    ],
)

Start by tracing the Rust request model, existing typed tool variants, tool registry, and Responses execution loop described in the issue. Inspect how response state, streaming, continuation, storage, and serialization are currently handled. Done means the listed wire shapes and action variants work through blocking and streaming flows, with isolated-handler safety and compatibility tests covering the acceptance criteria.