Question about multimodal embedding JSONL schema: handling empty images / positive_images / negative_images

Checklist / 检查清单

• I have searched existing issues, and this is a new question or discussion topic. / 我已经搜索过现有的 issues，确认这是一个新的问题与讨论。

Question Description / 问题描述

Hi, thanks for the great project.

I’m preparing JSONL training data for multimodal embedding and would like to clarify the expected data schema, especially for text-only samples.

A full multimodal sample looks like:
{ "messages": [{"role": "user", "content": "<image>sentence1"}], "images": ["/some/images.jpg"], "positive_messages": [[{"role": "user", "content": "<image>sentence2"}]], "positive_images": [["/some/positive_images.jpg"]], "negative_messages": [[{"role": "user", "content": "<image>sentence3"}]], "negative_images": [["/some/negative_images.jpg"]] }

However, in practice:
• Some messages / positive_messages / negative_messages are text-only
• Some samples include images

My questions are:

1. If messages contains no image tokens, should "images" be:
   • omitted, or
   • kept as "images": []?
2. For positive_messages / negative_messages:
   • If a candidate exists but is text-only, should *_images be [] or [[]] to keep alignment?
3. Is it required that all entries in a single JSONL file share the exact same set of keys, even if some values are empty?

I previously hit this error during loading:
All the data files must have the same columns, but at some point there are new columns ({'negative_images'})

Start by locating the JSONL loading path that emits “All the data files must have the same columns” and compare how messages, images, positive_images, and negative_images are parsed. Done means the expected handling of omitted, empty, and nested image fields is established for text-only and multimodal samples, with the schema guidance recorded.