Original file name only reaches the model on OpenAI/Azure — missing for Anthropic, Gemini, Bedrock, Ollama
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 200
- Forks
- 30
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 35
Description
Context
#348 added MediaMessageContent.file_name and populated it end to end, so every
image/document content object now carries the name of the file the user uploaded.
That part is provider-agnostic. The second half — actually rendering the name into
the request payload so the model can read it — was implemented for OpenAI and
Azure OpenAI only, via file_name_text_block() in flo_ai/llm/base_llm.py.
On every other provider the name is carried but never sent, so an agent asked to
report the filename of a document it was given still has no source for it and will
invent a plausible-looking one. That was the original bug #348 set out to fix; it is
only fixed for two of the seven providers.
Current state
| Provider | Image | Document |
|---|---|---|
OpenAI |
done | done |
AzureOpenAI |
done | done |
OpenAIVLLM (subclasses OpenAI) |
inherited | inherited |
Anthropic |
n/a — raises NotImplementedError |
missing |
Gemini |
missing | missing |
VertexAI (subclasses Gemini) |
missing | missing |
AWSBedrock |
missing | missing |
OllamaLLM |
n/a — raises NotImplementedError |
missing |
RootFloLLM |
delegates — inherits whatever the wrapped provider does | delegates |
Per-provider notes
Anthropic — cleanest of the set, and probably the one to do first. The Messages
API document block has a native title field, so no synthetic text block is needed:
set title in format_document_in_message (anthropic_llm.py:259). Images raise
NotImplementedError, so there's nothing to do there.
Gemini / VertexAI — the real work. format_image_in_message and
format_document_in_message each return a single types.Part, and
gemini_llm.generate appends msg['content'] straight into contents. Emitting a
name Part alongside the media Part means returning multiple Parts, which the content
assembly has to accept. Not a one-liner — worth confirming how the google-genai SDK
handles a nested list in contents before picking an approach.
AWSBedrock — format_image_in_message returns a bare dict rather than a list,
while documents fall through to BaseLLM._rasterize_pdf_to_images, which emits the
OpenAI block shape. Those two shapes are already inconsistent with each other,
independent of file names; worth untangling before adding anything.
Ollama — images raise NotImplementedError; documents use the same inherited
OpenAI-shaped rasterizer as Bedrock, so it has the same question.
Design decision to settle
BaseLLM._rasterize_pdf_to_images deliberately produces the OpenAI Chat Completions
shape and is inherited by Bedrock and Ollama. Adding the name block there would
cover both for free, but it would also mean the base class decides prompt content for
providers that never opted in. #348 avoided that on purpose by overriding
format_document_in_message in OpenAI/AzureOpenAI instead. Whichever way this
goes, it should be a deliberate call rather than a side effect.
Constraint to preserve
The name must ride on the same message as the media, never as a separate message.
An earlier attempt (#282) injected a standalone UserMessage and was removed in #319:
a ForEach over input_filter: [input] counts one item per input message, so one
extra message per file doubles the iterations and runs the per-item pipeline on bare
filename strings.
Acceptance criteria
- A named document/image produces a payload the model can read the file name from,
on Anthropic, Gemini, VertexAI, Bedrock and Ollama - Message count is unchanged — the name never becomes its own message
- Unnamed media produces byte-identical payloads to today
- Cached formatting (
_formatted_cache) is not mutated, so name blocks can't stack
across nodes and retries — see the regression test in
tests/unit-tests/test_openai_llm.py::test_openai_format_document_includes_file_name - Per-provider unit tests mirroring the three added in #348
Out of scope
Sanitising the file name before it enters the prompt (truncation, newline stripping).
It's user-controlled text going into a prompt on every provider, so it deserves its own
issue — but it's a pre-existing property of the approach, not a regression from #348.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with flo_ai/llm/base_llm.py and the provider formatting methods, then inspect anthropic_llm.py:259 and gemini_llm.generate to understand how media content reaches each request. Review the existing regression test in tests/unit-tests/test_openai_llm.py and add equivalent provider coverage. Done means named media carries its filename on the same message, unnamed payloads remain byte-identical, and cached formatting does not accumulate name blocks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, google-cloud, python
- Domain
- ai, backend-api-design, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100