spring-projects / spring-projects/spring-ai

Honor `MessagePart` order in `BedrockProxyChatModel` user messages

Open
#7,012 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 7h
Merged PRs (30d)
6

Description

Please do a quick search on GitHub issues first, the feature you are about to request might have already been requested.

Expected Behavior

BedrockProxyChatModel should build the Converse ContentBlock list of a user message in the order of UserMessage.getParts(), so that media can be placed before text:

UserMessage message = UserMessage.builder()                                                                                                                                        
      .part(MediaPart.of(video))                                                                                                                                                   
      .part(TextPart.of("Instructions and question about the video ..."))                                                                                                          
      .build();                                                                                                                                                                    

Expected Converse request content: [video, text].

Messages built with the existing text(...) / media(...) builder methods or the legacy constructors already get their parts in the legacy order (text, then media), so their
request output would not change.

Current Behavior

BedrockProxyChatModel.createRequest reads getText() and getMedia() separately and always emits the text block first, followed by all media blocks:

if (StringUtils.hasText(userMessage.getText())) {                                                                                                                                  
      contents.add(ContentBlock.fromText(userMessage.getText()));                                                                                                                  
}                                                                                                                                                                                  
                                                                                                                                                                                   
if (!CollectionUtils.isEmpty(userMessage.getMedia())) {                                                                                                                            
      List<ContentBlock> mediaContent = userMessage.getMedia()                                                                                                                     
              .stream()                                                                                                                                                            
              .map(this::mapMediaToContentBlock)                                                                                                                                   
              .toList();                                                                                                                                                           
      contents.addAll(mediaContent);                                                                                                                                               
}                                                                                                                                                                                  

Since #6997 a UserMessage carries an ordered List<MessagePart>, but the Bedrock Converse adapter does not consume it yet, so the order expressed by the caller is lost and the
message above is still sent as [text, video].

Context

The AWS prompting guidance for video understanding with Amazon Nova recommends placing the video before the text in the user turn, and putting the important instructions in the
user prompt rather than in the system prompt.
A video is tokenized into a large number of tokens, so instructions that come before it (system prompt, or text preceding the video) can get diluted.

https://docs.aws.amazon.com/nova/latest/userguide/prompting-vision-prompting.html

We recommend that you place media files (such as images or videos) before adding any documents, followed by your instructional text or prompts to guide the model. While images placed after text or interspersed with text will still perform adequately, if the use case permits, the {media_file}-then-{text} structure is the preferred approach.

I hit this in a video analysis use case.
Because the block order cannot be controlled through Spring AI, I had to drop BedrockProxyChatModel and call the AWS SDK ConverseRequest API directly, which means losing the
ChatModel abstraction, tool calling support, observability, etc. for that feature.

Proposed change, limited to spring-ai-bedrock-converse:

  • iterate over userMessage.getParts() in the USER branch, mapping TextPart to ContentBlock.fromText and MediaPart to the existing mapMediaToContentBlock
  • keep skipping empty text blocks (gh-6695) and keep the cache point as the last block
  • add unit tests asserting the content block order of the generated ConverseRequest for both explicit parts and legacy-built messages
  • document the ordering in the Bedrock Converse multimodality section of the reference docs

No new public API is needed.
ChatClient's fluent user(u -> u.text(...).media(...)) spec is intentionally out of scope here; callers can pass a UserMessage built with parts through
prompt().messages(...).

Related: #6997 introduced MessagePart; #6729 is an earlier proposal for ordered user content.

I would be happy to submit a pull request for this if the direction sounds good.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the spring-ai-bedrock-converse module at BedrockProxyChatModel.createRequest and inspect UserMessage.getParts(), the existing media mapper, and related unit tests. Verify generated ConverseRequest content block order for explicit parts and legacy-built messages, including empty text and cache-point behavior, then update the Bedrock Converse multimodality reference documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, spring
Domain
ai, api, documentation, testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.