spring-projects / spring-projects/spring-ai
It seems that LLM struggle to comprehend JSON schemas that include Java-type definitions?
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 9.5k
- Forks
- 2.9k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 6
Description
for example
@Bean
public ChatClient chatClient(OllamaChatModel ollamaChatModel) {
return ChatClient.builder(ollamaChatModel)
.defaultAdvisors(new MessageChatMemoryAdvisor(new InMemoryChatMemory()))
.build();
}
@Component
public class MatchPlanTool {
@Tool(name = "matchPlan", returnDirect = true, description = "Analyze the user's travel plan, which includes departure location, destination, trip start time(in the user's timezone), and number of travelers (including adults and children).")
public String matchPlan(Request request) {
return request.desc();
}
@Data
@JsonClassDescription("Analyze the user's travel plan service API request parameter")
public static class Request {
@JsonPropertyDescription("departure location")
private String departure;
@JsonPropertyDescription("destination")
private String destination;
@JsonPropertyDescription("trip start time")
private String tripStartTime;
@JsonPropertyDescription("number of adults")
private Integer adultCount;
@JsonPropertyDescription("number of children")
private Integer childrenCount;
public String desc() {
return String.format("user need %s time,from %s to %s,include adults %s,chlidren%s", tripStartTime, departure, destination, adultCount, childrenCount);
}
}
}
@GetMapping(value = "chat")
public String chat(@RequestParam String sessionId, @RequestParam String message) {
return chatClient.prompt()
.user(message)
.advisors(i -> i.param(AbstractChatMemoryAdvisor.CHAT_MEMORY_CONVERSATION_ID_KEY, sessionId))
.tools(matchPlanTool)
.call()
.content();
}
This is the JSON schema generated by Spring AI before calling LLM
{
"$schema" : "https://json-schema.org/draft/2020-12/schema",
"type" : "object",
"properties" : {
"request" : {
"type" : "object",
"properties" : {
"adults" : {
"type" : "integer",
"format" : "int32",
"description" : "number of adults"
},
"children" : {
"type" : "integer",
"format" : "int32",
"description" : "number of children"
},
"departure" : {
"type" : "string",
"description" : "departure location"
},
"destination" : {
"type" : "string",
"description" : "destination"
},
"tripStartTime" : {
"type" : "string",
"description" : "trip start time"
}
},
"required" : [ "adults", "children", "departure", "destination", "tripStartTime" ],
"description" : "Analyze the user's travel plan service API request parameter"
}
},
"required" : [ "request" ],
"additionalProperties" : false
}
It seems that LLM struggle to comprehend JSON schemas that include Java-type definitions?
LLM response is this
{"request":{"departureLocation":"beijing","destination":"shanghai","numAdults":2,"numChildren":1,"tripStartTime":"2023-12-25T10:00:00+08:00"}}
look at this, field name is not match, Causing JSON deserialization failure。
Did I make a mistake somewhere?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the ChatClient tool call with MatchPlanTool and compare the generated schema's request field names with the model's response. Trace how the Java Request properties become the tool schema and how the response is deserialized; done means identifying whether schema generation or model adherence causes the mismatch and documenting or correcting the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- ai, backend-api-design
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100