spring-projects / spring-projects/spring-ai
Support streaming (token-by-token) together with tool execution in ChatClient
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 9.5k
- Forks
- 2.9k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 5
Description
Description
Summary
When using ChatClient.prompt()... .stream().content() we get token-level streaming, but the tool-call loop (e.g. ToolCallAdvisor, tool execution, multi-turn tool calls) does not run on the stream path. When using .call(), tools run but the response is only available after the full completion, so we cannot stream tokens to the client. It would be very useful to have streaming and tool execution in the same flow so that apps can stream the final text to the client even when the model uses tools (e.g. agent skills, RAG, file read).
Current behavior (Spring AI 2.0.0-M2)
.call()uses the call advisor chain (e.g.getCallAdvisors(),nextCall()) and ToolCallAdvisor runs: tool calls are executed, results are sent back to the model, and the final response is returned once. No token-level streaming..stream()uses the stream advisor chain (getStreamAdvisors(),nextStream()) and terminates withChatModelStreamAdvisor. The ToolCallAdvisor is not in the stream path, so tool calls in the model response are not executed in a loop. We only get a stream of the first model reply (or no tool execution at all).
So we must choose: either streaming (good UX) or tool execution (skills/agents), but not both for the same request.
Expected behavior
- Option A: When using
.stream(), the same tool-call loop that runs on.call()could run in the background: when the model requests a tool call, execute it, append the tool response to the conversation, and continue the stream with the next model turn. The client would see tokens streamed after each turn (including after tool execution). - Option B: Or expose an advisor / hook so that the stream path can participate in tool execution (e.g.
ToolCallAdvisoror a new advisor that supports bothadviseCalland streaming, so the stream chain can execute tools and re-invoke the model until the final text is streamed).
Use case
We have a clinical report app that uses agent skills (e.g. MMPI-3 interpretation) and tools (e.g. read files, skills from .claude/skills). For “Interpret by domain” and “Interpret by test” we need both:
- Tool/skill execution (model can call Read, use skill context, etc.).
- Streaming the final narrative to the UI (token-by-token over SSE).
Today we use .call() for those two flows and send the full response in one SSE message, which works but loses the streaming UX. All other prompts use a separate ChatClient without tools and .stream().content() for chunked SSE.
Version
- Spring AI: 2.0.0-M2
- Spring Boot: 4.0.x (or whatever you use)
References (optional)
DefaultChatClient:.call()usesadvisorChain.getCallAdvisors()/nextCall();.stream()usesadvisorChain.getStreamAdvisors()/nextStream().- Tool execution (e.g.
DefaultToolCallingManager.executeToolCalls) is triggered from the call path; the stream path does not run this loop.
Thank you for considering this. Happy to provide more context or sample code if useful.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with DefaultChatClient and compare the call advisor chain using getCallAdvisors()/nextCall() with the stream chain using getStreamAdvisors()/nextStream(). Trace how DefaultToolCallingManager.executeToolCalls participates in the call path and how the stream path ends at ChatModelStreamAdvisor. Done means tool calls can execute across turns while the final model response remains available as token-level streaming.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring, spring-boot
- Domain
- ai, api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100