spring-projects / spring-projects/spring-ai
[Bedrock Converse] Place a `cachePoint` on the last tool result message during tool-calling rounds
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 9.5k
- Forks
- 2.9k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 6
Description
BedrockProxyChatModel never places a cache point on tool result messages, so during multi-round tool calling the tool outputs are billed as uncached input on every round.
This is the same gap that was reported for the Anthropic client in #6261 and fixed by #6268 (cacheToolResults on AnthropicCacheOptions). The Bedrock Converse client was not updated.
Current behaviour
In BedrockProxyChatModel.createRequest(), shouldApplyCachePoint is computed only against lastUserMessageIndex:
boolean shouldApplyCachePoint = shouldCacheConversationHistory && i == lastUserMessageIndex;
and it is consumed only inside the MessageType.USER branch:
if (shouldApplyCachePoint) {
contents.add(ContentBlock.fromCachePoint(buildCachePoint(cacheOptions)));
logger.debug("Applied cache point on last user message (conversation history caching)");
}
The MessageType.TOOL branch builds its ToolResultBlocks into a ConversationRole.USER message and never adds a ContentBlock.fromCachePoint(...):
else if (message.getMessageType() == MessageType.TOOL) {
List<ContentBlock> contentBlocks = new ArrayList<>(
((ToolResponseMessage) message).getResponses().stream().map(toolResponse -> {
ToolResultBlock toolResultBlock = ToolResultBlock.builder()
.toolUseId(toolResponse.id())
.content(ToolResultContentBlock.builder().text(toolResponse.responseData()).build())
.build();
return ContentBlock.fromToolResult(toolResultBlock);
}).toList());
instructionMessages.add(Message.builder().content(contentBlocks).role(ConversationRole.USER).build());
}
Why it matters
Each tool-calling round appends an assistant toolUse message and a user toolResult message, and the whole grown history is re-sent on the next round.
Because the only messages cache point sits on the last user message, everything appended after it — the assistant toolUse blocks and all toolResult blocks — falls outside the cached prefix.
With CONVERSATION_HISTORY, tool results are therefore reprocessed as uncached input on every round, which is expensive when tools return large payloads (the reporter of #6261 measured ~11.8k tokens of tool results billed uncached across a two-turn conversation).
Note that BedrockCacheOptions exposes only strategy, multiBlockSystemCaching and ttl, and no BedrockCacheStrategy variant targets tool results, so there is currently no way to opt in.
Expected behaviour
An opt-in equivalent of the Anthropic fix: place a cache point after the last toolResult block of the last tool result message in the request, so prior tool outputs are read from cache on subsequent rounds instead of being reprocessed.
The AWS Converse API allows this — cachePoint is accepted in the messages field for all Claude models on Bedrock (up to 4 checkpoints per request), and a tool result message is an ordinary user message.
A cacheToolResults flag on BedrockCacheOptions, taking effect with CONVERSATION_HISTORY, would mirror the Anthropic API and keep the two clients consistent — as was done for multiBlockSystemCaching (#6117 / #6250).
Side-note: It would be worth exposing the flag on BedrockCacheProperties too. The Anthropic cacheToolResults flag is currently programmatic-only and absent from AnthropicCacheProperties, so autoconfiguration users cannot reach it on either client.
Environment
- spring-ai-bedrock-converse
2.0.2-SNAPSHOT(currentmain)
See #6261
See #6268
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in BedrockProxyChatModel.createRequest(), then compare the existing Anthropic fix in #6268 and inspect BedrockCacheOptions and BedrockCacheProperties. Trace how CONVERSATION_HISTORY handles tool messages and define completion as an opt-in cacheToolResults setting that places the cache point after the final tool result while keeping the two clients consistent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- api, backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100