Streaming Tool Call Arguments Not Merged When vLLM Returns `id` in Every Chunk

Streaming Tool Call Arguments Not Merged When vLLM Returns id in Every Chunk

Bug description

When using Spring AI's OpenAI module with vLLM-deployed models in streaming mode, tool call arguments are not properly merged. This occurs because vLLM returns the same id field in every streaming chunk, which causes the merge logic in OpenAiStreamFunctionCallingHelper to incorrectly treat each chunk as a new tool call instead of merging the arguments.

The current merge logic uses StringUtils.hasText(currentToolCall.id()) to determine if a chunk is a new tool call. When each chunk contains the same id, this logic prevents argument merging, resulting in empty or null arguments being passed to tool execution.

According to OpenAI's official SDK implementation, the index field should be used to identify the same tool call, not the presence of id.

Environment

• Spring AI version: main branch (latest)
• Java version: 17+
• Module: spring-ai-openai
• Model deployment: vLLM (OpenAI-compatible inference engine)
• Mode: streaming enabled (stream=true)
• Tool calling: enabled
• Vector store: N/A

Steps to reproduce

1. Deploy a model with vLLM that supports function calling:

   vllm serve /path/to/model --tool-call-parser hermes --enable-auto-tool-choice
   

2. Configure Spring AI to connect to vLLM:

   OpenAiApi openAiApi = OpenAiApi.builder()
       .baseUrl("http://localhost:8000/v1")
       .apiKey("EMPTY")
       .build();
   
   OpenAiChatModel chatModel = OpenAiChatModel.builder()
       .openAiApi(openAiApi)
       .defaultOptions(OpenAiChatOptions.builder()
           .model("model-name")
           .build())
       .build();
   

3. Define a tool and use streaming mode:

   @Tool(description = "Get weather information")
   public String getWeather(@ToolParam(description = "City name") String city,
                            @ToolParam(description = "Date") String date) {
       return "Weather info for " + city + " on " + date;
   }
   
   ChatClient chatClient = ChatClient.builder(chatModel)
       .defaultToolCallbacks(MethodToolCallbackProvider.builder()
           .toolObjects(new WeatherTools())
           .build())
       .build();
   
   chatClient.prompt()
       .user("What's the weather in Beijing on 2024-01-01?")
       .stream()
       .content()
       .subscribe(System.out::println);
   

4. Observe the error:

   [WARN] Tool call arguments are null or empty for tool: get_weather. Using empty JSON object as default.
   [ERROR] IllegalStateException: No ToolCallback found for tool name: get_weather
   

5. Inspect the streaming response chunks (vLLM sends id in every chunk):

   • Chunk 1: {"index": 0, "id": "call_xxx", "function": {"name": "get_weather", "arguments": ""}}
   • Chunk 2: {"index": 0, "id": "call_xxx", "function": {"arguments": "{\"city\": \""}}
   • Chunk 3: {"index": 0, "id": "call_xxx", "function": {"arguments": "Beijing\", \"date"}}
   • ...more chunks with same index and id

Expected behavior

Streaming tool call chunks with the same index field should be merged into a complete tool call:

{
  "id": "call_xxx",
  "type": "function",
  "function": {
    "name": "get_weather",
    "arguments": "{\"city\": \"Beijing\", \"date\": \"2024-01-01\"}"
  }
}

The merge logic should:

• Use index field comparison to identify the same tool call (per OpenAI SDK standard)
• Concatenate arguments from all chunks with the same index
• Preserve name from the first chunk
• Work correctly regardless of whether id appears in every chunk or only the first chunk

Minimal Complete Reproducible example

@Test
public void testVllmStreamingToolCallMerge() {
    // Simulate vLLM streaming response where each chunk has the same id
    
    // Chunk 1: Tool call initialization
    var toolCall1 = new OpenAiApi.ChatCompletionMessage.ToolCall(0, "call_123", "function",
        new OpenAiApi.ChatCompletionMessage.ChatCompletionFunction("get_weather", ""));
    var delta1 = new OpenAiApi.ChatCompletionMessage(null, null, null, null, 
        List.of(toolCall1), null, null, null, null);
    var choice1 = new OpenAiApi.ChatCompletionChunk.ChunkChoice(null, 0, delta1, null);
    var chunk1 = new OpenAiApi.ChatCompletionChunk("id1", List.of(choice1), 1L, "model", 
        null, null, null, null);
    
    // Chunk 2: Same id, same index, partial arguments
    var toolCall2 = new OpenAiApi.ChatCompletionMessage.ToolCall(0, "call_123", "function",
        new OpenAiApi.ChatCompletionMessage.ChatCompletionFunction(null, "{\"city\": \""));
    var delta2 = new OpenAiApi.ChatCompletionMessage(null, null, null, null, 
        List.of(toolCall2), null, null, null, null);
    var choice2 = new OpenAiApi.ChatCompletionChunk.ChunkChoice(null, 0, delta2, null);
    var chunk2 = new OpenAiApi.ChatCompletionChunk("id1", List.of(choice2), 1L, "model", 
        null, null, null, null);
    
    // Chunk 3: Same id, same index, rest of arguments
    var toolCall3 = new OpenAiApi.ChatCompletionMessage.ToolCall(0, "call_123", "function",
        new OpenAiApi.ChatCompletionMessage.ChatCompletionFunction(null, "Beijing\"}"));
    var delta3 = new OpenAiApi.ChatCompletionMessage(null, null, null, null, 
        List.of(toolCall3), null, null, null, null);
    var choice3 = new OpenAiApi.ChatCompletionChunk.ChunkChoice(null, 0, delta3, null);
    var chunk3 = new OpenAiApi.ChatCompletionChunk("id1", List.of(choice3), 1L, "model", 
        null, null, null, null);
    
    // Merge chunks using OpenAiStreamFunctionCallingHelper
    var helper = new OpenAiStreamFunctionCallingHelper();
    var merged1 = helper.merge(null, chunk1);
    var merged2 = helper.merge(merged1, chunk2);
    var merged3 = helper.merge(merged2, chunk3);
    
    // Expected: arguments should be merged
    var mergedToolCall = merged3.choices().get(0).delta().toolCalls().get(0);
    
    // This assertion FAILS with current code (arguments is "" or null)
    assertThat(mergedToolCall.function().arguments())
        .isEqualTo("{\"city\": \"Beijing\"}");
    
    // This assertion PASSES with the fix
    assertThat(mergedToolCall.function().name()).isEqualTo("get_weather");
    assertThat(mergedToolCall.index()).isEqualTo(0);
}

Root cause

In OpenAiStreamFunctionCallingHelper.java (around line 129):

if (StringUtils.hasText(currentToolCall.id())) {
    // Treated as new tool call, arguments NOT merged
    toolCalls.add(currentToolCall);
}
else {
    toolCalls.add(merge(lastPreviousTooCall, currentToolCall));
}

The problem: When vLLM sends id in every chunk, each chunk is treated as a new tool call, preventing argument merging.

The fix: Use index field comparison instead:

boolean isSameToolCall = lastPreviousTooCall != null 
    && currentToolCall.index() != null
    && currentToolCall.index().equals(lastPreviousTooCall.index());

if (StringUtils.hasText(currentToolCall.id()) && !isSameToolCall) {
    toolCalls.add(currentToolCall);
} else {
    toolCalls.add(merge(lastPreviousTooCall, currentToolCall));
}

Also fix merge(ToolCall, ToolCall) to preserve index:

Integer index = (current.index() != null ? current.index() : previous.index());
return new ToolCall(index, id, type, function);

Impact

This issue affects:

• vLLM users - vLLM is a popular high-throughput LLM serving engine that returns id in every streaming chunk
• Other OpenAI-compatible inference engines that include id in every chunk for consistency
• Mock API testing where test data includes id in all chunks

Related issues

• #5806 - Streaming tool calls incorrectly merged (similar root cause)
• #4790 - Related discussion about same-ID merge handling
• #4629 - Tool Call chunk merging for empty string IDs (fixed)
• #2627 - Incorrect merge of ChatCompletionFunction (fixed)

Reference

OpenAI Node.js SDK implementation uses index field for tool call identification:
https://github.com/openai/openai-node/blob/master/src/lib/ChatCompletionStream.ts

Start in OpenAiStreamFunctionCallingHelper.java around line 129 and trace merge(ToolCall, ToolCall); use testVllmStreamingToolCallMerge as the focused reproducer. Check that chunks sharing an index retain the function name and concatenate arguments, including when every chunk has an id, and that the merged call preserves its index.