spring-projects / spring-projects/spring-ai

ChatClient streaming does not work with LM Studio Local LLM API Server

Open
#2,445 5 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 7h
Merged PRs (30d)
6

Description

Bug description
I ran LM Studio Local LLM API Server (OpenAI compatibility mode). ChatClient.call() API works perfectly, but the ChatClient.stream() does not work!

I enabled debug logs and also used the SimpleLoggerAdvisor and saw this log for streaming:

DEBUG 54629 --- [streaming] [nio-8080-exec-6] o.s.web.servlet.DispatcherServlet        : POST "/stream", parameters={}
DEBUG 54629 --- [streaming] [nio-8080-exec-6] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to dev.danvega.streaming.ChatController#chatWithStream(String)
DEBUG 54629 --- [streaming] [nio-8080-exec-6] m.m.a.RequestResponseBodyMethodProcessor : Read "application/json;charset=UTF-8" to ["{"message": "What is the largest country on Earth?"}"]
DEBUG 54629 --- [streaming] [nio-8080-exec-6] o.s.a.c.c.advisor.SimpleLoggerAdvisor    : request: AdvisedRequest[chatModel=OpenAiChatModel [defaultOptions=OpenAiChatOptions: {"streamUsage":false,"model":"meta-llama-3.1-8b-instruct","temperature":0.7}], userText={"message": "What is the largest country on Earth?"}, systemText=, chatOptions=OpenAiChatOptions: {"streamUsage":false,"model":"meta-llama-3.1-8b-instruct","temperature":0.7}, media=[], functionNames=[], functionCallbacks=[], messages=[], userParams={}, systemParams={}, advisors=[org.springframework.ai.chat.client.advisor.observation.ObservableRequestResponseAdvisor@6e5f4d0], advisorParams={}]
DEBUG 54629 --- [streaming] [nio-8080-exec-6] o.s.w.r.f.client.ExchangeFunctions       : [8220a7c] HTTP POST http://localhost:1234/v1/chat/completions
DEBUG 54629 --- [streaming] [nio-8080-exec-6] o.s.http.codec.json.Jackson2JsonEncoder  : [8220a7c] Encoding [ChatCompletionRequest[messages=[ChatCompletionMessage[rawContent={"message": "What is the largest co (truncated)...]
DEBUG 54629 --- [streaming] [nio-8080-exec-6] o.s.w.c.request.async.WebAsyncManager    : Started async request for "/stream"
DEBUG 54629 --- [streaming] [nio-8080-exec-6] o.s.web.servlet.DispatcherServlet        : Exiting but response remains open for further handling

But I could not see anything in the LM Studio server log!

Environment
Spring AI version: 1.0.0-M6
Java version: 21
OS: macOS and Windows 11

Steps to reproduce
I used the spring-ai-openai-spring-boot-starter lib with this configuration:

  ai:
    openai:
      api-key: lm-studio
      base-url: http://localhost:1234
      chat:
        options:
          model: meta-llama-3.1-8b-instruct
      embedding:
        options:
          model: text-embedding-nomic-embed-text-v1.5-embedding

Expected behavior
When I call LM Studio Local LLM API Server (OpenAI compatibility mode) using the curl command (with stream enabled), the ChatClient.call() API also works with the above configuration.

Minimal Complete Reproducible example
You can run an LM Studio Local LLM API Server (OpenAI compatibility mode) and clone this example from @danvega and apply my my configs. You will see that the ChatClient.call() works, but the stream() does not!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the ChatClient.stream() path in the spring-ai-openai-spring-boot-starter and compare it with the working ChatClient.call() path. Reproduce against LM Studio at http://localhost:1234/v1/chat/completions using the linked spring-ai-streaming example and the supplied configuration. Done means streaming requests reach LM Studio and return streamed output while the existing call() behavior remains working.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.