rootflo / rootflo/wavefront

Original file name only reaches the model on OpenAI/Azure — missing for Anthropic, Gemini, Bedrock, Ollama

Open
#349 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
200
Forks
30
Avg merge
1d 11h
Merged PRs (30d)
35

Description

Context

#348 added MediaMessageContent.file_name and populated it end to end, so every
image/document content object now carries the name of the file the user uploaded.

That part is provider-agnostic. The second half — actually rendering the name into
the request payload so the model can read it — was implemented for OpenAI and
Azure OpenAI only
, via file_name_text_block() in flo_ai/llm/base_llm.py.

On every other provider the name is carried but never sent, so an agent asked to
report the filename of a document it was given still has no source for it and will
invent a plausible-looking one. That was the original bug #348 set out to fix; it is
only fixed for two of the seven providers.

Current state

Provider Image Document
OpenAI done done
AzureOpenAI done done
OpenAIVLLM (subclasses OpenAI) inherited inherited
Anthropic n/a — raises NotImplementedError missing
Gemini missing missing
VertexAI (subclasses Gemini) missing missing
AWSBedrock missing missing
OllamaLLM n/a — raises NotImplementedError missing
RootFloLLM delegates — inherits whatever the wrapped provider does delegates

Per-provider notes

Anthropic — cleanest of the set, and probably the one to do first. The Messages
API document block has a native title field, so no synthetic text block is needed:
set title in format_document_in_message (anthropic_llm.py:259). Images raise
NotImplementedError, so there's nothing to do there.

Gemini / VertexAI — the real work. format_image_in_message and
format_document_in_message each return a single types.Part, and
gemini_llm.generate appends msg['content'] straight into contents. Emitting a
name Part alongside the media Part means returning multiple Parts, which the content
assembly has to accept. Not a one-liner — worth confirming how the google-genai SDK
handles a nested list in contents before picking an approach.

AWSBedrockformat_image_in_message returns a bare dict rather than a list,
while documents fall through to BaseLLM._rasterize_pdf_to_images, which emits the
OpenAI block shape. Those two shapes are already inconsistent with each other,
independent of file names; worth untangling before adding anything.

Ollama — images raise NotImplementedError; documents use the same inherited
OpenAI-shaped rasterizer as Bedrock, so it has the same question.

Design decision to settle

BaseLLM._rasterize_pdf_to_images deliberately produces the OpenAI Chat Completions
shape and is inherited by Bedrock and Ollama. Adding the name block there would
cover both for free, but it would also mean the base class decides prompt content for
providers that never opted in. #348 avoided that on purpose by overriding
format_document_in_message in OpenAI/AzureOpenAI instead. Whichever way this
goes, it should be a deliberate call rather than a side effect.

Constraint to preserve

The name must ride on the same message as the media, never as a separate message.
An earlier attempt (#282) injected a standalone UserMessage and was removed in #319:
a ForEach over input_filter: [input] counts one item per input message, so one
extra message per file doubles the iterations and runs the per-item pipeline on bare
filename strings.

Acceptance criteria

  • A named document/image produces a payload the model can read the file name from,
    on Anthropic, Gemini, VertexAI, Bedrock and Ollama
  • Message count is unchanged — the name never becomes its own message
  • Unnamed media produces byte-identical payloads to today
  • Cached formatting (_formatted_cache) is not mutated, so name blocks can't stack
    across nodes and retries — see the regression test in
    tests/unit-tests/test_openai_llm.py::test_openai_format_document_includes_file_name
  • Per-provider unit tests mirroring the three added in #348

Out of scope

Sanitising the file name before it enters the prompt (truncation, newline stripping).
It's user-controlled text going into a prompt on every provider, so it deserves its own
issue — but it's a pre-existing property of the approach, not a regression from #348.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with flo_ai/llm/base_llm.py and the provider formatting methods, then inspect anthropic_llm.py:259 and gemini_llm.generate to understand how media content reaches each request. Review the existing regression test in tests/unit-tests/test_openai_llm.py and add equivalent provider coverage. Done means named media carries its filename on the same message, unnamed payloads remain byte-identical, and cached formatting does not accumulate name blocks.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, google-cloud, python
Domain
ai, backend-api-design, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.