vllm-project / vllm-project/agentic-api

Support async function and custom tool calls across Responses continuations

Open
#332 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
284
Forks
74
Avg merge
1d 17h
Merged PRs (30d)
93

Description

Problem statement / motivation

Async tools let a model continue independent work after emitting a call and accept its output later on the original call_id. Existing parallel gateway execution waits for a round's tool calls before continuing inference. Function normalization also drops async, and typed function/custom call outputs do not preserve the marker.

Proposed solution

Add typed async declaration/call metadata and a pending-call continuation contract. Keep client-executed function/custom tools application-executed. Preserve the marker and call identity through JSON/SSE output, persistence, and replay. Decide explicitly which upstreams can produce async calls and which gateway-executed tools may opt into nonblocking execution; reject unsupported combinations.

Acceptance criteria
  • async: true survives declaration normalization and emitted call serialization for supported function/custom tools.
  • A reference scenario starts a slow tool, produces independent output, and submits the eventual tool call output on its original call_id in a later response.
  • Streaming exposes a complete call early enough for the application to dispatch it while consuming remaining output.
  • Continuations preserve unresolved calls and accept outputs arriving across responses; unknown, duplicate, and conflicting outputs have explicit behavior.
  • A synchronous wait-tool example returns completed outputs before wait status; the wait tool remains application-defined.
  • Gateway-managed pending work, if enabled, has entry/byte/time limits, cancellation, disconnect policy, and joined task teardown. Acknowledging an async declaration alone must not imply this execution support.
  • Record supported combinations with parallel calls and multi-agent mode; preserve ordinary synchronous tool behavior.
Additional context

Related: #181 (completed parallel scheduling), #298 (multi-agent), #241/#244 (streaming boundaries). Extend the shared typed ingestion and ordered delivery path; keep pending-work scheduling in orchestration and ingestion inline.

Use the existing cassette recorder/scenario workflow for OpenAI-reference and gateway recordings; do not hand-author captures.

Source: async tool calling.

Reviewed at cdac36f: implementation. Documentation checked September 18, 2026; pin the implementation-time API/model version when recording conformance.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with crates/agentic-server-core/src/tool/function.rs and the shared typed ingestion and ordered delivery path, then review orchestration for continuation and pending-work boundaries. Use the existing cassette recorder/scenario workflow for OpenAI-reference and gateway recordings; done means async metadata and call identity survive serialization, persistence, replay, streaming, and later responses while unsupported combinations and pending-work behavior are explicit.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.