spring-projects / spring-projects/spring-ai

ToolCallingAdvisor loses advisor-injected messages after a subsequent tool call in 2.0.1 streaming mode

Open
#6,842 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 7h
Merged PRs (30d)
6

Description

Bug description

ToolCallingAdvisor in Spring AI 2.0.1 loses messages added by an override of
doGetNextInstructionsForToolCallStream(...) when the model makes another tool call.

This breaks advisors that insert application feedback into the tool loop. For example, an
application advisor appends a UserMessage containing FOO_BAR_FEEDBACK after a successful
edit tool call. The model receives that feedback and then calls read. The next recursive
request no longer contains the injected message, so the model cannot act on that feedback.

This behavior does not occur with 2.0.0-M8, which preserves the injected message through the
next tool call.

Cause

Regression commit

The regression was introduced by
029af3796e0289c8e1a1bd73f5a4a1463f7469fb,
Add configurable tool call limits. The commit is included in v2.0.1.

Before this commit, the streaming loop executed tools with finalRequest.prompt(), which
contained the complete message list sent to the model. The commit introduced a separate
fullTurnHistory parameter to preserve untrimmed history for tool-call limit accounting.

In 2.0.1, the streaming implementation of ToolCallingAdvisor maintains two different
message lists:

  1. The current instructions list is sent to the model. It contains messages appended by an
    override of doGetNextInstructionsForToolCallStream(...).
  2. A separate fullTurnHistory list is passed to ToolCallingManager.executeToolCalls(...).
    It is initialized from the original instructions and then updated from
    ToolExecutionResult.conversationHistory().

After the hook adds a message to nextInstructions, it is not copied to fullTurnHistory.
The next tool result is therefore built from stale history. The injected message is visible for
exactly one model request and disappears after the next tool call.

Specifically, the recursive call receives nextInstructions containing the injected message but
receives toolExecutionResult.conversationHistory() as the next fullTurnHistory. On the next
tool call, fullTurnHistory is passed to ToolCallingManager.executeToolCalls(...) without the
injected message.

The regression is visible in the 2.0.1 implementation because internalStream(...) carries
both instructions and fullTurnHistory; the recursive path uses the latter for tool execution.
The 2.0.0-M8 implementation carries the current prompt messages through recursive tool
execution instead.

The issue is not related to the tool-call limit. The failing conversation has fewer than ten tool
calls; Spring AI 2.0.1 defaults to 150 total calls per turn.

Environment

  • Spring AI: 2.0.1 (affected)
  • Spring AI: 2.0.0-M8 (not affected)
  • Spring Boot: 4.1.0
  • Transport: OpenAI-compatible streaming chat completions through Azure AI Foundry
  • Tool calling: ToolCallingAdvisor with conversationHistoryEnabled = true

Steps to reproduce

  1. Configure a ChatClient with streaming and a subclass of ToolCallingAdvisor.
  2. Override doGetNextInstructionsForToolCallStream(...) to append a UserMessage after a
    successful edit tool result.
  3. Use a scripted chat model that produces the following rounds:
    1. edit
    2. read
    3. final text response
  4. Inspect the requests received by the scripted model.

Actual request histories with 2.0.1:

  1. Initial request.
  2. After edit: includes FOO_BAR_FEEDBACK.
  3. After read: FOO_BAR_FEEDBACK is absent.

The third request must retain the injected feedback because the model needs it to construct the
final response.

Expected behavior

Every recursive tool-call round must preserve the complete message list sent to the previous
model round, including messages injected by ToolCallingAdvisor subclasses. A message appended
by doGetNextInstructionsForToolCallStream(...) must still be present after the model makes one
or more subsequent tool calls.

Minimal Complete Reproducible example

LLM generated unit test, sorry

The following unverified JUnit test is an illustrative reproducer. It uses a mocked ChatModel
that records every prompt and emits edit, read, then a final text response.

package org.springframework.ai.chat.client.advisor;

import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.when;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.ChatClientRequest;
import org.springframework.ai.chat.client.ChatClientResponse;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.messages.Message;
import org.springframework.ai.chat.messages.UserMessage;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.model.Generation;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.ai.model.tool.DefaultToolCallingChatOptions;
import org.springframework.ai.model.tool.ToolCallingManager;
import org.springframework.ai.model.tool.ToolExecutionResult;
import org.springframework.ai.tool.annotation.Tool;
import reactor.core.publisher.Flux;

class ToolCallingAdvisorMessageHistoryTests {

  @Test
  void preservesMessagesInjectedBetweenToolCalls() {
    var model = org.mockito.Mockito.mock(ChatModel.class);
    var prompts = new ArrayList<List<Message>>();
    var responses = List.of(toolCall("edit"), toolCall("read"), text("done"));
    var responseIndex = new AtomicInteger();

    when(model.getOptions()).thenReturn(DefaultToolCallingChatOptions.builder().build());
    when(model.stream(any(Prompt.class))).thenAnswer(invocation -> {
      prompts.add(List.copyOf(invocation.<Prompt>getArgument(0).getInstructions()));
      return Flux.just(responses.get(responseIndex.getAndIncrement()));
    });

    ChatClient.builder(model)
        .defaultTools(new Tools())
        .defaultAdvisors(new FeedbackAdvisor())
        .build()
        .prompt("update the document")
        .stream()
        .content()
        .blockLast();

    // Request 2 is generated after edit and correctly contains the injected message.
    assertThat(texts(prompts.get(1))).contains("FOO_BAR_FEEDBACK");

    // Fails on 2.0.1: the read round reconstructs history without the injected message.
    assertThat(texts(prompts.get(2))).contains("FOO_BAR_FEEDBACK");
  }

  private static final class FeedbackAdvisor extends ToolCallingAdvisor {

    private final AtomicBoolean feedbackAdded = new AtomicBoolean();

    private FeedbackAdvisor() {
      super(
          ToolCallingManager.builder().build(),
          DEFAULT_TOOL_EXECUTION_ELIGIBILITY_CHECKER,
          DEFAULT_ORDER,
          true);
    }

    @Override
    protected List<Message> doGetNextInstructionsForToolCallStream(
        ChatClientRequest request,
        ChatClientResponse response,
        ToolExecutionResult result) {
      var messages = new ArrayList<>(
          super.doGetNextInstructionsForToolCallStream(request, response, result));
      if (feedbackAdded.compareAndSet(false, true)) {
        messages.add(new UserMessage("FOO_BAR_FEEDBACK"));
      }
      return messages;
    }
  }

  private static final class Tools {

    @Tool
    public String edit() {
      return "ok";
    }

    @Tool
    public String read() {
      return "ok";
    }
  }

  private static ChatResponse toolCall(String name) {
    var call = new AssistantMessage.ToolCall("call-" + name, "function", name, "{}");
    var message = AssistantMessage.builder().toolCalls(List.of(call)).build();
    return new ChatResponse(List.of(new Generation(message)));
  }

  private static ChatResponse text(String value) {
    return new ChatResponse(List.of(new Generation(new AssistantMessage(value))));
  }

  private static List<String> texts(List<Message> messages) {
    return messages.stream().map(Message::getText).toList();
  }
}

Suggested fix

Keep fullTurnHistory for tool-call limit accounting, but merge into it any messages added by
doGetNextInstructionsForToolCallStream(...) before the next tool execution. The history used to
build the next round must include all messages returned by that hook.

A focused regression test should append a UserMessage in that hook, execute another tool call,
and verify that the following model request still contains the injected message.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in ToolCallingAdvisor, focusing on internalStream(...) and the recursive path that passes fullTurnHistory to ToolCallingManager.executeToolCalls(...). Add a focused regression test based on the supplied reproducer: append FOO_BAR_FEEDBACK in doGetNextInstructionsForToolCallStream(...), produce edit then read tool calls, and verify the following model request still contains the message.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring
Domain
api, backend, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.