spring-projects / spring-projects/spring-ai
ToolCallingAdvisor loses advisor-injected messages after a subsequent tool call in 2.0.1 streaming mode
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 9.5k
- Forks
- 2.9k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 6
Description
Bug description
ToolCallingAdvisor in Spring AI 2.0.1 loses messages added by an override of
doGetNextInstructionsForToolCallStream(...) when the model makes another tool call.
This breaks advisors that insert application feedback into the tool loop. For example, an
application advisor appends a UserMessage containing FOO_BAR_FEEDBACK after a successful
edit tool call. The model receives that feedback and then calls read. The next recursive
request no longer contains the injected message, so the model cannot act on that feedback.
This behavior does not occur with 2.0.0-M8, which preserves the injected message through the
next tool call.
Cause
Regression commit
The regression was introduced by
029af3796e0289c8e1a1bd73f5a4a1463f7469fb,
Add configurable tool call limits. The commit is included in v2.0.1.
Before this commit, the streaming loop executed tools with finalRequest.prompt(), which
contained the complete message list sent to the model. The commit introduced a separate
fullTurnHistory parameter to preserve untrimmed history for tool-call limit accounting.
In 2.0.1, the streaming implementation of ToolCallingAdvisor maintains two different
message lists:
- The current
instructionslist is sent to the model. It contains messages appended by an
override ofdoGetNextInstructionsForToolCallStream(...). - A separate
fullTurnHistorylist is passed toToolCallingManager.executeToolCalls(...).
It is initialized from the original instructions and then updated from
ToolExecutionResult.conversationHistory().
After the hook adds a message to nextInstructions, it is not copied to fullTurnHistory.
The next tool result is therefore built from stale history. The injected message is visible for
exactly one model request and disappears after the next tool call.
Specifically, the recursive call receives nextInstructions containing the injected message but
receives toolExecutionResult.conversationHistory() as the next fullTurnHistory. On the next
tool call, fullTurnHistory is passed to ToolCallingManager.executeToolCalls(...) without the
injected message.
The regression is visible in the 2.0.1 implementation because internalStream(...) carries
both instructions and fullTurnHistory; the recursive path uses the latter for tool execution.
The 2.0.0-M8 implementation carries the current prompt messages through recursive tool
execution instead.
The issue is not related to the tool-call limit. The failing conversation has fewer than ten tool
calls; Spring AI 2.0.1 defaults to 150 total calls per turn.
Environment
- Spring AI:
2.0.1(affected) - Spring AI:
2.0.0-M8(not affected) - Spring Boot:
4.1.0 - Transport: OpenAI-compatible streaming chat completions through Azure AI Foundry
- Tool calling:
ToolCallingAdvisorwithconversationHistoryEnabled = true
Steps to reproduce
- Configure a
ChatClientwith streaming and a subclass ofToolCallingAdvisor. - Override
doGetNextInstructionsForToolCallStream(...)to append aUserMessageafter a
successfuledittool result. - Use a scripted chat model that produces the following rounds:
editread- final text response
- Inspect the requests received by the scripted model.
Actual request histories with 2.0.1:
- Initial request.
- After
edit: includesFOO_BAR_FEEDBACK. - After
read:FOO_BAR_FEEDBACKis absent.
The third request must retain the injected feedback because the model needs it to construct the
final response.
Expected behavior
Every recursive tool-call round must preserve the complete message list sent to the previous
model round, including messages injected by ToolCallingAdvisor subclasses. A message appended
by doGetNextInstructionsForToolCallStream(...) must still be present after the model makes one
or more subsequent tool calls.
Minimal Complete Reproducible example
LLM generated unit test, sorry
The following unverified JUnit test is an illustrative reproducer. It uses a mocked ChatModel
that records every prompt and emits edit, read, then a final text response.
package org.springframework.ai.chat.client.advisor;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.when;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.ChatClientRequest;
import org.springframework.ai.chat.client.ChatClientResponse;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.messages.Message;
import org.springframework.ai.chat.messages.UserMessage;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.model.Generation;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.ai.model.tool.DefaultToolCallingChatOptions;
import org.springframework.ai.model.tool.ToolCallingManager;
import org.springframework.ai.model.tool.ToolExecutionResult;
import org.springframework.ai.tool.annotation.Tool;
import reactor.core.publisher.Flux;
class ToolCallingAdvisorMessageHistoryTests {
@Test
void preservesMessagesInjectedBetweenToolCalls() {
var model = org.mockito.Mockito.mock(ChatModel.class);
var prompts = new ArrayList<List<Message>>();
var responses = List.of(toolCall("edit"), toolCall("read"), text("done"));
var responseIndex = new AtomicInteger();
when(model.getOptions()).thenReturn(DefaultToolCallingChatOptions.builder().build());
when(model.stream(any(Prompt.class))).thenAnswer(invocation -> {
prompts.add(List.copyOf(invocation.<Prompt>getArgument(0).getInstructions()));
return Flux.just(responses.get(responseIndex.getAndIncrement()));
});
ChatClient.builder(model)
.defaultTools(new Tools())
.defaultAdvisors(new FeedbackAdvisor())
.build()
.prompt("update the document")
.stream()
.content()
.blockLast();
// Request 2 is generated after edit and correctly contains the injected message.
assertThat(texts(prompts.get(1))).contains("FOO_BAR_FEEDBACK");
// Fails on 2.0.1: the read round reconstructs history without the injected message.
assertThat(texts(prompts.get(2))).contains("FOO_BAR_FEEDBACK");
}
private static final class FeedbackAdvisor extends ToolCallingAdvisor {
private final AtomicBoolean feedbackAdded = new AtomicBoolean();
private FeedbackAdvisor() {
super(
ToolCallingManager.builder().build(),
DEFAULT_TOOL_EXECUTION_ELIGIBILITY_CHECKER,
DEFAULT_ORDER,
true);
}
@Override
protected List<Message> doGetNextInstructionsForToolCallStream(
ChatClientRequest request,
ChatClientResponse response,
ToolExecutionResult result) {
var messages = new ArrayList<>(
super.doGetNextInstructionsForToolCallStream(request, response, result));
if (feedbackAdded.compareAndSet(false, true)) {
messages.add(new UserMessage("FOO_BAR_FEEDBACK"));
}
return messages;
}
}
private static final class Tools {
@Tool
public String edit() {
return "ok";
}
@Tool
public String read() {
return "ok";
}
}
private static ChatResponse toolCall(String name) {
var call = new AssistantMessage.ToolCall("call-" + name, "function", name, "{}");
var message = AssistantMessage.builder().toolCalls(List.of(call)).build();
return new ChatResponse(List.of(new Generation(message)));
}
private static ChatResponse text(String value) {
return new ChatResponse(List.of(new Generation(new AssistantMessage(value))));
}
private static List<String> texts(List<Message> messages) {
return messages.stream().map(Message::getText).toList();
}
}
Suggested fix
Keep fullTurnHistory for tool-call limit accounting, but merge into it any messages added by
doGetNextInstructionsForToolCallStream(...) before the next tool execution. The history used to
build the next round must include all messages returned by that hook.
A focused regression test should append a UserMessage in that hook, execute another tool call,
and verify that the following model request still contains the injected message.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in ToolCallingAdvisor, focusing on internalStream(...) and the recursive path that passes fullTurnHistory to ToolCallingManager.executeToolCalls(...). Add a focused regression test based on the supplied reproducer: append FOO_BAR_FEEDBACK in doGetNextInstructionsForToolCallStream(...), produce edit then read tool calls, and verify the following model request still contains the message.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring
- Domain
- api, backend, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100