mudler / mudler/vllm.cpp

dots3-note: resized_size's detail / image_details / target_size overrides are not wired to the processor

Open
#2,645 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: MODEL-MM-dots3-note-dots3-note-for-causal-lm

Upstream's Dots3NoteImageProcessor.resized_size takes five inputs this port
does not read (vllm/models/dots3_note/common/processor.py:97-119 @
9035151d6):

    def resized_size(
        self, width, height, *,
        detail: str = "auto",
        min_pixels: int | None = None,
        max_pixels: int | None = None,
        target_height: int | None = None,
        target_width: int | None = None,
    ) -> tuple[int, int]:
        detail_config = self.image_details.get(detail, {})
        min_pixels = int(min_pixels if min_pixels is not None
                         else detail_config.get("min_pixels", self.min_pixels))
        max_pixels = int(max_pixels if max_pixels is not None
                         else detail_config.get("max_pixels", self.max_pixels))
        height = int(target_height or detail_config.get("target_height") or height)
        width = int(target_width or detail_config.get("target_width") or width)

What the port does

Dots3NoteResizedSize(height, width, factor, min_pixels, max_pixels) mirrors
the body of that function exactly, and LoadDots3NoteProcessorConfig reads the
min_pixels / max_pixels pair from preprocessor_config.json. What is
missing is the resolution ABOVE the body:

  • the detail string, which upstream takes per request,
  • the self.image_details[detail] override table, a Dots3NoteProcessor
    constructor argument, and
  • the explicit target_height / target_width arguments.

RouteDots3NoteImageRgb (src/vllm/entrypoints/openai/mm_chat_dots3note.cpp)
calls ProcessImage(rgb, height, width) with the decoded image's own size and
nothing else, so no path exists for any of the three to arrive.

Why it is not urgent, and why it is still owed

No released preprocessor_config.json for dots-studio/dots3-note-prev carries
an image_details table, and the default detail resolves to exactly the
config pair this port already reads, so the resolved geometry is identical for
every checkpoint the row can feed today. The gap is invisible rather than
absent: a checkpoint that shipped the table, or a request that set
image_url.detail, would be resized at the wrong budget with no refusal in
front of it.

What closing this needs

  • The OpenAI image_url part's detail field parsed and threaded through
    MakeDots3NoteImageChatFn -> RouteDots3NoteImageRgb -> ProcessImage.
  • image_details read from preprocessor_config.json into
    Dots3NoteProcessorConfig, with the same precedence upstream uses
    (explicit argument, then detail table, then the top-level value).
  • A gate that shows a request naming a detail whose table entry differs from
    the top-level pair resolves to the table's geometry.

Record

Owner: this row, W8 (the multimodal front end, which owns the request parsing;
.agents/specs/dots3-note.md §7). Recorded under ## Owed in that spec.
Found while landing W6c (#2537), which ported the resampler itself and left
this argument-resolution half explicitly out of scope rather than quietly
widening into W8.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with vllm/models/dots3_note/common/processor.py, Dots3NoteProcessorConfig, and the request path through MakeDots3NoteImageChatFn, RouteDots3NoteImageRgb, and ProcessImage. Trace how image_url.detail could reach resizing, then inspect preprocessor_config.json handling and .agents/specs/dots3-note.md §7. Done means detail and image_details precedence are threaded through and the described geometry gate passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
api, computer-vision
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.