[BUG][Mage-VL] `images=` + `videos=` in one processor call misaligns `<|image_pad|>` placeholders (codec raises, frames silently wrong)

Ran into this while wiring Mage-VL into a local pipeline that passes a couple of reference photos alongside a video clip. Passing images= and videos= to the same processor(...) call produces a prompt whose <|image_pad|> placeholders do not line up with the visual tensors returned next to them.

Two failure modes:

• video_backend="codec" — the reference images lose their placeholders entirely, and generate() dies with ValueError: Image features and image tokens do not match.
• video_backend="frames" (default) — the totals happen to agree, so nothing raises, the model's own n_image_tokens check passes, and you get a plausible-looking answer that was grounded on the wrong features. This one is the more dangerous of the two.

I think this is meant to work rather than being an unsupported combination: the __call__ docstring says images "May coexist with videos; expansion order in the prompt is determined by the chat_template / placeholders" (processing_mage_vl.py L217-218), and there is an explicit # If videos and images coexist, prefer concatenation of patch tensors branch at L454. The README only shows images-only and videos-only examples, so nothing there contradicts it.

Note the affected files (processing_mage_vl.py, codec_video_processing_mage_vl.py) ship as remote code in the HF repo rather than under mage_vl/ here — filing on the code repo, but happy to move this to the HF discussions tab if that's the right venue.

Environment

Minimal reproduction

Uses only files shipped in the repo (examples/dog.jpg, examples/soccer-broadcast.mp4).

import sys
import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM

MODEL_ID = sys.argv[1] if len(sys.argv) > 1 else "microsoft/Mage-VL"
IMAGE = f"{MODEL_ID}/examples/dog.jpg"
VIDEO = f"{MODEL_ID}/examples/soccer-broadcast.mp4"

processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
PAD_ID = processor.tokenizer.convert_tokens_to_ids("<|image_pad|>")
MERGE = processor.spatial_merge_size ** 2

messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "video"},
    {"type": "text", "text": "What animal is in the reference photo, and what happens in the video?"},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

inputs = processor(
    text=[text],
    images=[Image.open(IMAGE).convert("RGB")],
    videos=[VIDEO],
    video_backend="codec",
    codec_config={"engine": "hevc", "target_canvas": 12, "group_size": 32,
                  "images_per_group": 4, "patch": 16},
    max_pixels=150000,
    return_tensors="pt",
    padding=True,
)

n_pad = int((inputs["input_ids"][0] == PAD_ID).sum())
n_tok = int(sum(int(t) * int(h) * int(w) // MERGE for t, h, w in inputs["image_grid_thw"]))
print(f"<|image_pad|> placeholders in prompt : {n_pad}")
print(f"visual tokens from image_grid_thw    : {n_tok}")

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, trust_remote_code=True, dtype=torch.bfloat16,
    attn_implementation="sdpa").to("cuda:0").eval()
inputs = {k: (v.to("cuda:0") if hasattr(v, "to") else v) for k, v in inputs.items()}
inputs["pixel_values"] = inputs["pixel_values"].to(model.dtype)
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=64, do_sample=False)

Output:

<|image_pad|> placeholders in prompt : 3775
visual tokens from image_grid_thw    : 3776
ValueError: Image features and image tokens do not match: tokens: 3775, features 3776

Expected: the number of <|image_pad|> placeholders equals sum(t*h*w // spatial_merge_size**2) over image_grid_thw, and the k-th run of placeholders corresponds to the k-th row group of pixel_values / image_grid_thw.

Codec backend: the image placeholders are deleted

All numbers below are with repo assets: examples/dog.jpg (2048 visual tokens) and assets/mage-vl-cover.png (1508), plus examples/soccer-broadcast.mp4 at max_pixels=150000, codec_config={"engine": "hevc", "target_canvas": 12, "group_size": 32, "images_per_group": 4, "patch": 16} (12 canvases × 144 = 1728 tokens).

Two things stand out. The delta is exactly the number of images, not their token cost; and moving the images before or after the video in the prompt changes nothing — which is itself diagnostic, since everything between the first <|vision_start|> and the last <|vision_end|> is being replaced wholesale.

Comparing the run-length layout of <|image_pad|> in the video-only prompt against the images+video prompt shows where the budget went — they are identical except for the very first timestamp run:

video only        : 68 runs, first 5 = [144, 3, 2, 1, 29], total 1728
2 images + video  : 68 runs, first 5 = [3698, 3, 2, 1, 29], total 5282
identical after run 0: True

3698 = 144 + (2048 − 1) + (1508 − 1). The two reference images get no placeholder span of their own; their budget is grafted onto the video's first timestamp, and each image consumes one of the video's placeholders in the process (one placeholder replaced by n placeholders is a net gain of n−1), which is where the −2 comes from.

Frames backend: silent misalignment, no exception

Same two images, 8 sampled frames (510 tokens per frame). "prompt runs" is the run-length sequence of <|image_pad|> in the prompt; "grid rows" is the per-row token count from image_grid_thw.

• Images-first: the prompt is expanded correctly, but the tensors are stacked video-rows-first, so every placeholder run is paired with the wrong features.
• Video-first: the two reference images are left with one placeholder token each, and the first frame's run swells to 4064 = 510 + 2047 + 1507 — their budget is again taken out of the video.

In both cases the totals agree by coincidence (−2 consumed video pads, +2 leftover un-expanded image pads), so nothing raises.

Root cause

1. codec_video_processing_mage_vl.py::rewrite_text_with_codec_positions (L217-231)

vision_text = "".join(parts)
first_vs, last_ve = text.find(VISION_START), text.rfind(VISION_END)
if first_vs == -1 or last_ve == -1:
    return text
tail_start = last_ve + len(VISION_END)
if tail_start < len(text) and text[tail_start] == "\n":
    tail_start += 1
return text[:first_vs] + vision_text + text[tail_start:]

find(VISION_START) / rfind(VISION_END) span from the first vision block in the prompt to the last one, which silently assumes the video is the only visual block in the text. Once images= is also passed, the image placeholders sit inside that span and get replaced along with the video block. The frames backend does the analogous rewrite correctly — it regex-matches a single <|vision_start|>\s*<|video_pad|>\s*<|vision_end|> block — so only the codec path has this particular defect.

2. processing_mage_vl.py::__call__ — ordering (affects both backends)

• Expansion order. The video branches run first and rewrite the video block into literal runs of <|image_pad|> (codec: L306; frames: _expand_video_block_for_frames). The IMAGE PATH block then runs _expand_image_pads (L442), whose while IMAGE_PAD in s: s.replace(IMAGE_PAD, ..., 1) always restarts from the beginning of the string — so by that point the earliest <|image_pad|> may belong to the video, and the image expansion eats video placeholders.

• Tensor order. The concatenation at L454-458 is unconditionally video-rows-then-image-rows:

  # If videos and images coexist, prefer concatenation of patch tensors.
  if "pixel_values" in out:
      out["pixel_values"] = torch.cat([out["pixel_values"], image_outputs["pixel_values"]], dim=0)
  

  The model consumes <|image_pad|> slots in prompt order and pixel_values in row order, so this is only correct when the video precedes every image in the prompt — which is exactly what the docstring says is not assumed.

Suggested fix

(a) Rewrite only the video placeholder block — mirrors what the frames backend already does:

--- a/codec_video_processing_mage_vl.py
+++ b/codec_video_processing_mage_vl.py
@@
 import os
+import re
 import shutil
@@
 IMAGE_PAD = "<|image_pad|>"
+VIDEO_PAD = "<|video_pad|>"
+
+# Matches exactly one chat-template video placeholder block.
+_VIDEO_BLOCK_RE = re.compile(
+    re.escape(VISION_START) + r"\s*" + re.escape(VIDEO_PAD) + r"\s*" + re.escape(VISION_END)
+)
@@ def rewrite_text_with_codec_positions
     vision_text = "".join(parts)
-    first_vs, last_ve = text.find(VISION_START), text.rfind(VISION_END)
-    if first_vs == -1 or last_ve == -1:
+    match = _VIDEO_BLOCK_RE.search(text)
+    if match is None:
         return text
-    tail_start = last_ve + len(VISION_END)
+    tail_start = match.end()
     if tail_start < len(text) and text[tail_start] == "\n":
         tail_start += 1
-    return text[:first_vs] + vision_text + text[tail_start:]
+    return text[:match.start()] + vision_text + text[tail_start:]

The \n handling is unchanged, so video-only output stays byte-identical.

(b) Expand images before the video rewrite, and concatenate visuals in prompt order. Record the placeholder order before any rewriting; move the IMAGE PATH block ahead of the video branches (at that point every <|image_pad|> still belongs to a real image, because videos are still <|video_pad|>); have both video branches emit per-video slots instead of writing out directly; then assemble:

_pad_re = re.compile(re.escape(IMAGE_PAD) + "|" + re.escape(VIDEO_PAD))
order_per_text = [
    ["image" if m.group(0) == IMAGE_PAD else "video" for m in _pad_re.finditer(s)]
    for s in text
]
...
if image_slots or video_slots:
    flat_order = [k for per_text in order_per_text for k in per_text]
    if (sum(k == "image" for k in flat_order) == len(image_slots)
            and sum(k == "video" for k in flat_order) == len(video_slots)):
        img_it, vid_it = iter(image_slots), iter(video_slots)
        ordered = [next(img_it) if k == "image" else next(vid_it) for k in flat_order]
    else:                       # legacy fallback, not reachable via the chat template
        ordered = list(video_slots) + list(image_slots)
    out["pixel_values"] = torch.cat([s[0] for s in ordered], dim=0)
    out["image_grid_thw"] = torch.cat([s[1] for s in ordered], dim=0)
    out["patch_positions"] = torch.cat([s[2] for s in ordered], dim=0)

(When the codec branch replicates a single text across several videos, order_per_text has to be replicated with it.)

Reordering the stack is safe for the vision tower: _build_cu_seqlens / _build_block_attention_mask block strictly per image_grid_thw row, so image rows and video rows never attend to each other regardless of where they sit.

@@ -237,6 +237,64 @@
 
         out: dict = {}
 
+        # ---------------- VISUAL PLACEHOLDER ORDER ----------------
+        # Record the prompt order of every visual placeholder *before* any
+        # rewriting happens, so the visual tensors can later be concatenated in
+        # exactly the order the model will consume the <|image_pad|> slots.
+        _pad_re = re.compile(re.escape(IMAGE_PAD) + "|" + re.escape(VIDEO_PAD))
+        order_per_text = [
+            ["image" if m.group(0) == IMAGE_PAD else "video" for m in _pad_re.finditer(s)]
+            for s in text
+        ]
+
+        # ---------------- IMAGE PATH ----------------
+        # Runs *before* the video paths: at this point every <|image_pad|> in
+        # `text` still belongs to a real image, because the video placeholders
+        # are still <|video_pad|>. (Running it afterwards would let the image
+        # expansion eat placeholders the video rewrite had just emitted.)
+        image_slots: List[tuple] = []
+        video_slots: List[tuple] = []
+        if images is not None:
+            if self.image_processor is None:
+                raise ValueError("images passed but no image_processor configured.")
+            image_outputs = self.image_processor(images=images, return_tensors="pt")
+            image_grid_thw = image_outputs["image_grid_thw"]
+
+            sms = self.spatial_merge_size
+            merge_factor = sms * sms
+            image_token_counts = (
+                (image_grid_thw[:, 0] * image_grid_thw[:, 1] * image_grid_thw[:, 2])
+                // merge_factor
+            ).tolist()
+            img_idx = 0
+
+            def _expand_image_pads(s: str) -> str:
+                nonlocal img_idx
+                while IMAGE_PAD in s:
+                    if img_idx >= len(image_token_counts):
+                        break
+                    n = int(image_token_counts[img_idx])
+                    s = s.replace(IMAGE_PAD, "<|placeholder|>" * n, 1)
+                    img_idx += 1
+                return s.replace("<|placeholder|>", IMAGE_PAD)
+
+            text = [_expand_image_pads(s) for s in text]
+
+            try:
+                from .video_processing_mage_vl import build_patch_positions
+            except ImportError:
+                from video_processing_mage_vl import build_patch_positions
+            image_pp = build_patch_positions(image_grid_thw, spatial_merge_size=sms)
+            offset = 0
+            for row in image_grid_thw:
+                n = int(row[0]) * int(row[1]) * int(row[2])
+                image_slots.append((
+                    image_outputs["pixel_values"][offset: offset + n],
+                    row.unsqueeze(0),
+                    image_pp[offset: offset + n],
+                ))
+                offset += n
+
         # ---------------- CODEC VIDEO BACKEND ----------------
@@ -284,6 +342,7 @@
             if len(rewritten_texts) != len(videos_list):
                 if len(rewritten_texts) == 1 and len(videos_list) >= 1:
                     rewritten_texts = rewritten_texts * len(videos_list)
+                    order_per_text = order_per_text * len(videos_list)
                 else:
@@ -311,9 +370,7 @@
                 all_grid_thw.append(image_grid_thw)
                 all_patch_positions.append(patch_positions)
 
-            out["pixel_values"] = torch.cat(all_pixel_values, dim=0)
-            out["image_grid_thw"] = torch.cat(all_grid_thw, dim=0)
-            out["patch_positions"] = torch.cat(all_patch_positions, dim=0)
+            video_slots.extend(zip(all_pixel_values, all_grid_thw, all_patch_positions))
             text = rewritten_texts
@@ -412,68 +469,36 @@
-            out["pixel_values"] = video_outputs["pixel_values_videos"]
             vgthw = video_outputs["video_grid_thw"]
-            expanded_rows = []
+            video_pv = video_outputs["pixel_values_videos"]
+            video_pp = video_outputs["patch_positions"]
+            offset = 0
             for row in vgthw:
                 T_v, H_v, W_v = int(row[0]), int(row[1]), int(row[2])
-                expanded_rows.extend([[1, H_v, W_v]] * T_v)
-            out["image_grid_thw"] = torch.tensor(expanded_rows, dtype=vgthw.dtype)
-            out["patch_positions"] = video_outputs["patch_positions"]
-
-        # ---------------- IMAGE PATH ----------------
-        if images is not None:
-            ...   # (the whole old IMAGE PATH block moves up, unchanged except
-            ...   #  for emitting per-image slots instead of writing `out`)
+                n = T_v * H_v * W_v
+                video_slots.append((
+                    video_pv[offset: offset + n],
+                    torch.tensor([[1, H_v, W_v]] * T_v, dtype=vgthw.dtype),
+                    video_pp[offset: offset + n],
+                ))
+                offset += n
+
+        # ---------------- ASSEMBLE VISUALS IN PROMPT ORDER ----------------
+        if image_slots or video_slots:
+            flat_order = [kind for per_text in order_per_text for kind in per_text]
+            n_img = sum(k == "image" for k in flat_order)
+            n_vid = sum(k == "video" for k in flat_order)
+            if n_img == len(image_slots) and n_vid == len(video_slots):
+                img_it, vid_it = iter(image_slots), iter(video_slots)
+                ordered = [next(img_it) if k == "image" else next(vid_it) for k in flat_order]
+            else:
+                # Placeholder bookkeeping did not line up (e.g. a caller-supplied
+                # prompt that bypassed the chat template). Fall back to the legacy
+                # videos-then-images concatenation rather than failing hard.
+                ordered = list(video_slots) + list(image_slots)
+            out["pixel_values"] = torch.cat([s[0] for s in ordered], dim=0)
+            out["image_grid_thw"] = torch.cat([s[1] for s in ordered], dim=0)
+            out["patch_positions"] = torch.cat([s[2] for s in ordered], dim=0)

Verification

No regression on single-modality paths — same prompts, stock vs patched processor, tensors compared with torch.equal:

[codec video-only]  input_ids=True pixel_values=True image_grid_thw=True patch_positions=True
[frames video-only] input_ids=True pixel_values=True image_grid_thw=True patch_positions=True
[image-only]        input_ids=True pixel_values=True image_grid_thw=True patch_positions=True
[2 images only]     input_ids=True pixel_values=True image_grid_thw=True patch_positions=True

Mixed images + video line up in both backends and both prompt orders:

[codec/image-first]   pads=5284 tokens=5284 -> MATCH   grid rows [2048, 1508, 144, 144, ...]
[codec/video-first]   pads=5284 tokens=5284 -> MATCH   grid rows [..., 144, 144, 2048, 1508]
[frames/image-first]  pads=7636 tokens=7636 -> MATCH   runs == grid rows elementwise: True
[frames/video-first]  pads=7636 tokens=7636 -> MATCH   runs == grid rows elementwise: True

For the frames backend, runs == grid rows is now true element-by-element, not just in total — the k-th placeholder run is the k-th tensor row group.

Generation, patched processor + unmodified weights, examples/dog.jpg + examples/soccer-broadcast.mp4, prompt [video, image, text], question "What animal is in the reference photo, and what happens in the video?":

• codec backend (3776 / 3776, previously a hard ValueError): "A dog is in the reference photo, and the video shows a sports broadcast with four commentators discussing a soccer match."
• frames backend (6128 / 6128): "A dog is in the reference photo, and the video shows a football match between England and Argentina."

One unrelated observation

With the placeholder accounting exact in every case, answer quality still depends noticeably on where the image sits in the prompt: with [video, image, text] the model answers both halves of the question, while with [image, video, text] it replies with just dog and ignores the video — in both backends. Counts are exact either way, so this looks like a prompt-distribution thing rather than a processing bug; mentioning it in case it is of interest.

Also not addressed by the patch: the codec branch replicates a single text once per video (rewritten_texts = rewritten_texts * len(videos_list)), so several videos inside one prompt is unsupported before and after — each video gets its own prompt in the batch.

Happy to open a PR against whichever repo is the source of truth for these files.

Start with processing_mage_vl.py::call and codec_video_processing_mage_vl.py::rewrite_text_with_codec_positions, then run the minimal reproduction with the supplied images, video, and both backends. Compare placeholder runs with image_grid_thw and pixel_values order; done means counts match, each placeholder run maps to the corresponding visual row, and video-only output remains unchanged.