vllm-project / vllm-project/agentic-api
Support async function and custom tool calls across Responses continuations
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 284
- Forks
- 74
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 93
Description
Problem statement / motivation
Async tools let a model continue independent work after emitting a call and accept its output later on the original call_id. Existing parallel gateway execution waits for a round's tool calls before continuing inference. Function normalization also drops async, and typed function/custom call outputs do not preserve the marker.
Proposed solution
Add typed async declaration/call metadata and a pending-call continuation contract. Keep client-executed function/custom tools application-executed. Preserve the marker and call identity through JSON/SSE output, persistence, and replay. Decide explicitly which upstreams can produce async calls and which gateway-executed tools may opt into nonblocking execution; reject unsupported combinations.
Acceptance criteria
-
async: truesurvives declaration normalization and emitted call serialization for supported function/custom tools. - A reference scenario starts a slow tool, produces independent output, and submits the eventual tool call output on its original
call_idin a later response. - Streaming exposes a complete call early enough for the application to dispatch it while consuming remaining output.
- Continuations preserve unresolved calls and accept outputs arriving across responses; unknown, duplicate, and conflicting outputs have explicit behavior.
- A synchronous wait-tool example returns completed outputs before wait status; the wait tool remains application-defined.
- Gateway-managed pending work, if enabled, has entry/byte/time limits, cancellation, disconnect policy, and joined task teardown. Acknowledging an async declaration alone must not imply this execution support.
- Record supported combinations with parallel calls and multi-agent mode; preserve ordinary synchronous tool behavior.
Additional context
Related: #181 (completed parallel scheduling), #298 (multi-agent), #241/#244 (streaming boundaries). Extend the shared typed ingestion and ordered delivery path; keep pending-work scheduling in orchestration and ingestion inline.
Use the existing cassette recorder/scenario workflow for OpenAI-reference and gateway recordings; do not hand-author captures.
Source: async tool calling.
Reviewed at cdac36f: implementation. Documentation checked September 18, 2026; pin the implementation-time API/model version when recording conformance.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with crates/agentic-server-core/src/tool/function.rs and the shared typed ingestion and ordered delivery path, then review orchestration for continuation and pending-work boundaries. Use the existing cassette recorder/scenario workflow for OpenAI-reference and gateway recordings; done means async metadata and call identity survive serialization, persistence, replay, streaming, and later responses while unsupported combinations and pending-work behavior are explicit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100