spring-projects / spring-projects/spring-ai

[Bedrock Converse] Place a `cachePoint` on the last tool result message during tool-calling rounds

Open
#6,938 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 7h
Merged PRs (30d)
6

Description

BedrockProxyChatModel never places a cache point on tool result messages, so during multi-round tool calling the tool outputs are billed as uncached input on every round.

This is the same gap that was reported for the Anthropic client in #6261 and fixed by #6268 (cacheToolResults on AnthropicCacheOptions). The Bedrock Converse client was not updated.

Current behaviour

In BedrockProxyChatModel.createRequest(), shouldApplyCachePoint is computed only against lastUserMessageIndex:

boolean shouldApplyCachePoint = shouldCacheConversationHistory && i == lastUserMessageIndex;

and it is consumed only inside the MessageType.USER branch:

if (shouldApplyCachePoint) {
    contents.add(ContentBlock.fromCachePoint(buildCachePoint(cacheOptions)));
    logger.debug("Applied cache point on last user message (conversation history caching)");
}

The MessageType.TOOL branch builds its ToolResultBlocks into a ConversationRole.USER message and never adds a ContentBlock.fromCachePoint(...):

else if (message.getMessageType() == MessageType.TOOL) {
    List<ContentBlock> contentBlocks = new ArrayList<>(
            ((ToolResponseMessage) message).getResponses().stream().map(toolResponse -> {
                ToolResultBlock toolResultBlock = ToolResultBlock.builder()
                    .toolUseId(toolResponse.id())
                    .content(ToolResultContentBlock.builder().text(toolResponse.responseData()).build())
                    .build();
                return ContentBlock.fromToolResult(toolResultBlock);
            }).toList());

    instructionMessages.add(Message.builder().content(contentBlocks).role(ConversationRole.USER).build());
}
Why it matters

Each tool-calling round appends an assistant toolUse message and a user toolResult message, and the whole grown history is re-sent on the next round.
Because the only messages cache point sits on the last user message, everything appended after it — the assistant toolUse blocks and all toolResult blocks — falls outside the cached prefix.
With CONVERSATION_HISTORY, tool results are therefore reprocessed as uncached input on every round, which is expensive when tools return large payloads (the reporter of #6261 measured ~11.8k tokens of tool results billed uncached across a two-turn conversation).

Note that BedrockCacheOptions exposes only strategy, multiBlockSystemCaching and ttl, and no BedrockCacheStrategy variant targets tool results, so there is currently no way to opt in.

Expected behaviour

An opt-in equivalent of the Anthropic fix: place a cache point after the last toolResult block of the last tool result message in the request, so prior tool outputs are read from cache on subsequent rounds instead of being reprocessed.

The AWS Converse API allows this — cachePoint is accepted in the messages field for all Claude models on Bedrock (up to 4 checkpoints per request), and a tool result message is an ordinary user message.

A cacheToolResults flag on BedrockCacheOptions, taking effect with CONVERSATION_HISTORY, would mirror the Anthropic API and keep the two clients consistent — as was done for multiBlockSystemCaching (#6117 / #6250).

Side-note: It would be worth exposing the flag on BedrockCacheProperties too. The Anthropic cacheToolResults flag is currently programmatic-only and absent from AnthropicCacheProperties, so autoconfiguration users cannot reach it on either client.

Environment
  • spring-ai-bedrock-converse 2.0.2-SNAPSHOT (current main)

See #6261
See #6268

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in BedrockProxyChatModel.createRequest(), then compare the existing Anthropic fix in #6268 and inspect BedrockCacheOptions and BedrockCacheProperties. Trace how CONVERSATION_HISTORY handles tool messages and define completion as an opt-in cacheToolResults setting that places the cache point after the final tool result while keeping the two clients consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
api, backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.