aws / aws/nova-prompt-optimizer

add support for OCR/Vision Prompting

Open
#51 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
56
Forks
9
PR merge metrics
No merged PRs in 30d

Description

To add vision/OCR support, you'd need to modify at minimum:

1. BedrockConverseHandler._get_messages() — Support image content blocks alongside text:
# Bedrock Converse API expects this for images:
{"image": {"format": "png", "source": {"bytes": }}}
2. PromptAdapter — Add a way to specify that a template variable should be populated with image data
rather than text.
3. DatasetAdapter — Handle dataset rows where an input column contains image file paths, loading the
bytes at inference time.
4. InferenceAdapter — Thread image data through to the Bedrock call.

Contributor guide

Open the contributing guide

Research direction

Start by tracing BedrockConverseHandler._get_messages(), PromptAdapter, DatasetAdapter, and InferenceAdapter to understand how text inputs move to the Bedrock call. Define how image paths become bytes and how image data is represented through each adapter, using the shown Converse API image block as the target. Done means dataset image inputs reach Bedrock alongside text for OCR or vision prompting.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
ai, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.