spring-projects / spring-projects/spring-ai

Support streaming (token-by-token) together with tool execution in ChatClient

Open
#5,616 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 10h
Merged PRs (30d)
5

Description

Description

Summary

When using ChatClient.prompt()... .stream().content() we get token-level streaming, but the tool-call loop (e.g. ToolCallAdvisor, tool execution, multi-turn tool calls) does not run on the stream path. When using .call(), tools run but the response is only available after the full completion, so we cannot stream tokens to the client. It would be very useful to have streaming and tool execution in the same flow so that apps can stream the final text to the client even when the model uses tools (e.g. agent skills, RAG, file read).

Current behavior (Spring AI 2.0.0-M2)
  • .call() uses the call advisor chain (e.g. getCallAdvisors(), nextCall()) and ToolCallAdvisor runs: tool calls are executed, results are sent back to the model, and the final response is returned once. No token-level streaming.
  • .stream() uses the stream advisor chain (getStreamAdvisors(), nextStream()) and terminates with ChatModelStreamAdvisor. The ToolCallAdvisor is not in the stream path, so tool calls in the model response are not executed in a loop. We only get a stream of the first model reply (or no tool execution at all).

So we must choose: either streaming (good UX) or tool execution (skills/agents), but not both for the same request.

Expected behavior
  • Option A: When using .stream(), the same tool-call loop that runs on .call() could run in the background: when the model requests a tool call, execute it, append the tool response to the conversation, and continue the stream with the next model turn. The client would see tokens streamed after each turn (including after tool execution).
  • Option B: Or expose an advisor / hook so that the stream path can participate in tool execution (e.g. ToolCallAdvisor or a new advisor that supports both adviseCall and streaming, so the stream chain can execute tools and re-invoke the model until the final text is streamed).
Use case

We have a clinical report app that uses agent skills (e.g. MMPI-3 interpretation) and tools (e.g. read files, skills from .claude/skills). For “Interpret by domain” and “Interpret by test” we need both:

  1. Tool/skill execution (model can call Read, use skill context, etc.).
  2. Streaming the final narrative to the UI (token-by-token over SSE).

Today we use .call() for those two flows and send the full response in one SSE message, which works but loses the streaming UX. All other prompts use a separate ChatClient without tools and .stream().content() for chunked SSE.

Version
  • Spring AI: 2.0.0-M2
  • Spring Boot: 4.0.x (or whatever you use)
References (optional)
  • DefaultChatClient: .call() uses advisorChain.getCallAdvisors() / nextCall(); .stream() uses advisorChain.getStreamAdvisors() / nextStream().
  • Tool execution (e.g. DefaultToolCallingManager.executeToolCalls) is triggered from the call path; the stream path does not run this loop.

Thank you for considering this. Happy to provide more context or sample code if useful.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with DefaultChatClient and compare the call advisor chain using getCallAdvisors()/nextCall() with the stream chain using getStreamAdvisors()/nextStream(). Trace how DefaultToolCallingManager.executeToolCalls participates in the call path and how the stream path ends at ChatModelStreamAdvisor. Done means tool calls can execute across turns while the final model response remains available as token-level streaming.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring, spring-boot
Domain
ai, api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.