spring-projects / spring-projects/spring-ai

ChatClient Vision Capability Fails Without Explicit CallAdvisor - java.lang.IllegalStateException: No CallAdvisors available to execute

Open
#3,302 3 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 7h
Merged PRs (30d)
6

Description

Bug description
Getting this error when trying to access the vision capability using ChatClient:
java.lang.IllegalStateException: No CallAdvisors available to execute

Environment

Steps to reproduce

  1. Clone the multimodal-llms project from this link.
  2. In VisionController.java, comment out this line:
    defaultAdvisors
  3. Start the app by running MultimodalLLMApplication.java:
    MultimodalLLMApplication.java
  4. Hit the following curl command:
    curl --request POST \
      --url http://localhost:8083/springai/v2/vision \
      --header 'Content-Type: multipart/form-data' \
      --form file=@/Users/dilipbandlasundarraj/Dilip/code-with-dilip/spring-ai/multimodal-llms/src/main/resources/files/vision/pizza.png \
      --form 'prompt=Explain what do you see in this picture?. Summarize it like an attractive menu item name.'
    
    

Expected behavior
The vision capability should work even without explicitly adding an advisor in the ChatClient.

Minimal Complete Reproducible example
The issue can be reproduced using the multimodal-llms sample project.
Comment out this line:
VisionController.java#L39
Then hit the POST /springai/v2/vision endpoint.

Fix
This endpoint works only when the following advisor is explicitly added in the controller constructor:

public VisionController(ChatClient.Builder chatClientBuilder,
                        OpenAiChatModel openAiChatModel) {
    this.openAiChatModel = openAiChatModel;
    var advisor = ChatModelCallAdvisor.builder()
            .chatModel(openAiChatModel)
            .build();
    this.chatClient = chatClientBuilder
            .defaultAdvisors(advisor)
            .build();
}

Source: VisionController.java#L39

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with multimodal-llms/src/main/java/com/llm/vision/VisionController.java, especially the defaultAdvisors configuration, and review MultimodalLLMApplication.java for application setup. Run the POST /springai/v2/vision curl reproduction with and without the explicit ChatModelCallAdvisor. Done means the vision endpoint works without explicitly adding that advisor.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring
Domain
ai, api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.