spring-projects / spring-projects/spring-ai

Unable to retrieve the reasoning content from the OpenAiChatModel during streaming calls

Open
#6,736 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 7h
Merged PRs (30d)
6

Description

Bug description
When chatting with the model, although the thinking switch is turned on, the thinking content still cannot be read from the response.


Image

As shown in the figure above, the additionalProperties.reasoning_content field has a value in the raw response content.

After investigation, I found the following issue:
the OpenAiChatModel#getReasoningContent method reads reasoning-content from ChatCompletionMessage#_additionalProperties, but no value is set for the message in OpenAiChatModel.ChunkMerge#chunkToChatCompletion, so the reasoning content cannot be retrieved.

Environment

  • Spring AI: 2.0.0
  • Java: 25

Steps to reproduce

  1. Configure OpenAI and turn on the thinking switch
spring:
  ai:
    openai:
      base-url: **
      api-key: **
      chat:
        model: **
        extra-body:
          enable_thinking: true
  1. Controller
package cn.com.deloitte.amsagent.ai.controller;

import io.swagger.v3.oas.annotations.Operation;
import lombok.*;
import lombok.extern.slf4j.Slf4j;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.http.MediaType;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import reactor.core.publisher.Flux;

import java.util.Objects;

@RestController
@RequestMapping("/chat")
@RequiredArgsConstructor
@Slf4j
public class ChatController {

    private final ChatClient chatClient;

    @PostMapping(value = "/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    public Flux<Message> stream(@RequestParam String question, @RequestParam(required = false) String conversationId) {
        return this.chatClient.prompt()
                .user(question)
                .stream()
                .chatResponse()
                .map(response -> {
                    Message msg = response.getResults().stream().map(result -> {
                        AssistantMessage output = result.getOutput();
                        Message.MessageBuilder builder = Message.builder().content(output.getText());
                        String reasoning =  (String) output.getMetadata().get("reasoningContent");
                        if (reasoning == null || reasoning.isEmpty()) {
                            // always true
                            return builder.build();
                        }

                        // never
                        log.info("reasoning: {}", reasoning);
                        builder.reasoningContent(reasoning);

                        return builder.build();

                    }).reduce(Message::merge).orElse(Message.builder().build());


                    // the reasoningContent is always null
                    log.info("reasoningContent: {}", msg.getReasoningContent());
                    if (!Objects.isNull(msg.getReasoningContent())) {
                        // never
                        throw new IllegalArgumentException("The reasoningContent has non-null value.");
                    }

                    return msg;
                });
    }


    @Data
    @Builder
    @NoArgsConstructor
    @AllArgsConstructor
    public static class Message {

        private String content;

        private String reasoningContent;

        public Message merge(Message msg) {
            if (content == null) {
                content = msg.content;
            } else if (msg.getContent() != null) {
                content = content + msg.content;
            }

            if (reasoningContent == null) {
                reasoningContent = msg.reasoningContent;
            } else if (msg.reasoningContent != null) {
                reasoningContent = reasoningContent + msg.reasoningContent;
            }
            return this;
        }

    }
}

  1. Start the application

  2. Call the api
    curl -X POST "http://127.0.0.1:8080/chat/stream?question=Why%20there%20isnot%20any%20thinking%20content?"
    Any more questions

Expected behavior
I expect an IllegalArgumentException to be thrown when the API is called. However, no matter how I modify thequestion parameter, the exception is never thrown.

Minimal Complete Reproducible example

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with OpenAiChatModel#getReasoningContent and OpenAiChatModel.ChunkMerge#chunkToChatCompletion, then trace how additionalProperties from streaming responses reaches ChatCompletionMessage. Reproduce the issue with enable_thinking enabled and verify that reasoning content is exposed through the response metadata during streaming calls.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.