spring-projects / spring-projects/spring-ai
Honor `MessagePart` order in `BedrockProxyChatModel` user messages
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 9.5k
- Forks
- 2.9k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 6
Description
Please do a quick search on GitHub issues first, the feature you are about to request might have already been requested.
Expected Behavior
BedrockProxyChatModel should build the Converse ContentBlock list of a user message in the order of UserMessage.getParts(), so that media can be placed before text:
UserMessage message = UserMessage.builder()
.part(MediaPart.of(video))
.part(TextPart.of("Instructions and question about the video ..."))
.build();
Expected Converse request content: [video, text].
Messages built with the existing text(...) / media(...) builder methods or the legacy constructors already get their parts in the legacy order (text, then media), so their
request output would not change.
Current Behavior
BedrockProxyChatModel.createRequest reads getText() and getMedia() separately and always emits the text block first, followed by all media blocks:
if (StringUtils.hasText(userMessage.getText())) {
contents.add(ContentBlock.fromText(userMessage.getText()));
}
if (!CollectionUtils.isEmpty(userMessage.getMedia())) {
List<ContentBlock> mediaContent = userMessage.getMedia()
.stream()
.map(this::mapMediaToContentBlock)
.toList();
contents.addAll(mediaContent);
}
Since #6997 a UserMessage carries an ordered List<MessagePart>, but the Bedrock Converse adapter does not consume it yet, so the order expressed by the caller is lost and the
message above is still sent as [text, video].
Context
The AWS prompting guidance for video understanding with Amazon Nova recommends placing the video before the text in the user turn, and putting the important instructions in the
user prompt rather than in the system prompt.
A video is tokenized into a large number of tokens, so instructions that come before it (system prompt, or text preceding the video) can get diluted.
https://docs.aws.amazon.com/nova/latest/userguide/prompting-vision-prompting.html
We recommend that you place media files (such as images or videos) before adding any documents, followed by your instructional text or prompts to guide the model. While images placed after text or interspersed with text will still perform adequately, if the use case permits, the {media_file}-then-{text} structure is the preferred approach.
I hit this in a video analysis use case.
Because the block order cannot be controlled through Spring AI, I had to drop BedrockProxyChatModel and call the AWS SDK ConverseRequest API directly, which means losing the
ChatModel abstraction, tool calling support, observability, etc. for that feature.
Proposed change, limited to spring-ai-bedrock-converse:
- iterate over
userMessage.getParts()in theUSERbranch, mappingTextParttoContentBlock.fromTextandMediaPartto the existingmapMediaToContentBlock - keep skipping empty text blocks (gh-6695) and keep the cache point as the last block
- add unit tests asserting the content block order of the generated
ConverseRequestfor both explicit parts and legacy-built messages - document the ordering in the Bedrock Converse multimodality section of the reference docs
No new public API is needed.
ChatClient's fluent user(u -> u.text(...).media(...)) spec is intentionally out of scope here; callers can pass a UserMessage built with parts through
prompt().messages(...).
Related: #6997 introduced MessagePart; #6729 is an earlier proposal for ordered user content.
I would be happy to submit a pull request for this if the direction sounds good.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in the spring-ai-bedrock-converse module at BedrockProxyChatModel.createRequest and inspect UserMessage.getParts(), the existing media mapper, and related unit tests. Verify generated ConverseRequest content block order for explicit parts and legacy-built messages, including empty text and cache-point behavior, then update the Bedrock Converse multimodality reference documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, spring
- Domain
- ai, api, documentation, testing
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100