crmne / crmne/ruby_llm

[FEATURE] Per-attachment media fidelity (Gemini media_resolution, OpenAI image detail)

Open
#888 1 comment 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Ruby
Stars
4.4k
Forks
504
Avg merge
1d 8h
Merged PRs (30d)
11

Description

Scope check
  • This is core LLM communication (not application logic)
  • This benefits most users (not just my use case)
  • This can't be solved in application code with current RubyLLM
  • I read the Contributing Guide
Due diligence
  • I searched existing issues
  • I checked the documentation
What problem does this solve?

Providers expose a per-media fidelity dial that directly trades tokens for extraction quality, and RubyLLM currently has no way to reach it:

  • Gemini accepts media_resolution per part: 280 / 560 / 1120 / 2240 tokens per image for LOW / MEDIUM / HIGH / ULTRA_HIGH, an 8x cost difference. generationConfig.mediaResolution covers the request-wide case and is already reachable via with_provider_options, but per-part settings override it and ULTRA_HIGH is only valid per part (docs), so the interesting cases (max fidelity at all; mixed fidelity in one request, e.g. page images at ULTRA_HIGH next to PDFs at HIGH) are unreachable.
  • OpenAI has the same concept spelled detail on input_image in the Responses API.

with_provider_options can't get there: it deep-merges into the payload, and parts live inside the contents array that the protocol Media modules build. Today the only way is prepending patches into Protocols::Gemini::Media / Protocols::Responses::Media (which is what we are currently doing).

Proposed solution

An attachment-level shared concept, in the spirit of transcription's task_type (shared concepts in signatures, provider spellings in protocols):

chat.ask("Where is the revenue figure?", with: RubyLLM::Attachment.new(io, filename: "page-3.png", resolution: :ultra_high))
  • Gemini media: emit media_resolution: { level: "MEDIA_RESOLUTION_ULTRA_HIGH" } on that part
  • OpenAI Responses media: map to detail: on input_image
  • Providers without the concept: ignore it
  • Unset: today's behavior, untouched

Naming could of course be different (resolution / fidelity / detail?), as is whether unsupported values should raise or drop silently. If you'd rather not grow Attachment, an alternative is a documented per-part passthrough, but it fits the deep-merge contract of provider_options poorly, which is why I thought attachment-level works better.

Why this belongs in RubyLLM

Part rendering happens inside the protocol Media modules, so application code can only monkeypatch them which we would like to stop doing :)

Happy to contribute the PR (both protocols, specs, docs) once you've steered the shape, we run this in production today and can validate against live extraction workloads.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading RubyLLM::Attachment and the existing part-rendering entry points in Protocols::Gemini::Media and Protocols::Responses::Media, along with the provider_options behavior. Confirm the attachment-level API with maintainers, then trace the relevant specs and documentation areas. Done means per-attachment fidelity reaches both provider payloads, unsupported providers remain unaffected, unset attachments keep today's behavior, and specs and docs cover the result.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
ai, api, backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.