spring-projects / spring-projects/spring-ai

It seems that LLM struggle to comprehend JSON schemas that include Java-type definitions?

Open
#2,763 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 7h
Merged PRs (30d)
6

Description

for example

@Bean
public ChatClient chatClient(OllamaChatModel ollamaChatModel) {
    return ChatClient.builder(ollamaChatModel)
            .defaultAdvisors(new MessageChatMemoryAdvisor(new InMemoryChatMemory()))
            .build();
}
@Component
public class MatchPlanTool {
    @Tool(name = "matchPlan", returnDirect = true, description = "Analyze the user's travel plan, which includes departure location, destination, trip start time(in the user's timezone), and number of travelers (including adults and children).")
    public String matchPlan(Request request) {
        return request.desc();
    }

    @Data
    @JsonClassDescription("Analyze the user's travel plan service API request parameter")
    public static class Request {

        @JsonPropertyDescription("departure location")
        private String departure;

        @JsonPropertyDescription("destination")
        private String destination;

        @JsonPropertyDescription("trip start time")
        private String tripStartTime;

        @JsonPropertyDescription("number of adults")
        private Integer adultCount;

        @JsonPropertyDescription("number of children")
        private Integer childrenCount;

        public String desc() {
            return String.format("user need %s time,from %s  to %s,include adults %s,chlidren%s", tripStartTime, departure, destination, adultCount, childrenCount);
        }
    }
}
  @GetMapping(value = "chat")
  public String chat(@RequestParam String sessionId, @RequestParam String message) {
      return chatClient.prompt()
              .user(message)
              .advisors(i -> i.param(AbstractChatMemoryAdvisor.CHAT_MEMORY_CONVERSATION_ID_KEY, sessionId))
              .tools(matchPlanTool)
              .call()
              .content();
  }

This is the JSON schema generated by Spring AI before calling LLM

{
  "$schema" : "https://json-schema.org/draft/2020-12/schema",
  "type" : "object",
  "properties" : {
    "request" : {
      "type" : "object",
      "properties" : {
        "adults" : {
          "type" : "integer",
          "format" : "int32",
          "description" : "number of adults"
        },
        "children" : {
          "type" : "integer",
          "format" : "int32",
          "description" : "number of children"
        },
        "departure" : {
          "type" : "string",
          "description" : "departure location"
        },
        "destination" : {
          "type" : "string",
          "description" : "destination"
        },
        "tripStartTime" : {
          "type" : "string",
          "description" : "trip start time"
        }
      },
      "required" : [ "adults", "children", "departure", "destination", "tripStartTime" ],
      "description" : "Analyze the user's travel plan service API request parameter"
    }
  },
  "required" : [ "request" ],
  "additionalProperties" : false
}

It seems that LLM struggle to comprehend JSON schemas that include Java-type definitions?
LLM response is this

{"request":{"departureLocation":"beijing","destination":"shanghai","numAdults":2,"numChildren":1,"tripStartTime":"2023-12-25T10:00:00+08:00"}}

look at this, field name is not match, Causing JSON deserialization failure。

Did I make a mistake somewhere?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the ChatClient tool call with MatchPlanTool and compare the generated schema's request field names with the model's response. Trace how the Java Request properties become the tool schema and how the response is deserialized; done means identifying whether schema generation or model adherence causes the mismatch and documenting or correcting the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
ai, backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.