ml-explore / ml-explore/mlx-swift-examples

LMInput restricts model input to a single collection of images and video frames

Open
#282 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Swift
Stars
2.7k
Forks
426
PR merge metrics
No merged PRs in 30d

Description

See #277 and #276

The UserInput struct can represent a series of messages with media attached to each image:

        return UserInput(
            chat: [
                .system(generate.system),
                .user(prompt, images: media.images, videos: media.videos),
            ],
            processing: media.processing
        )

This could include back and forth between the user and assistant including adding additional media.

The UserInputProcessor converts this to an LMInput:

public struct LMInput {
    public let text: Text
    public let image: ProcessedImage?
    public let video: ProcessedVideo?

but that only allows for one set of image/video. This should probably have:

    public let images: [ProcessedImage]
    public let videos: [ProcessedVideo]

though the model would have to be updated to take advantage of that.

Consider this chat:

> /image /tmp/img.jpeg


> what animal is in the image?
[["role": "system", "content": [["text": "You are a helpful assistant who answers questions in English.", "type": "text"]]], ["role": "user", "content": [["text": "what animal is in the image?", "type": "text"], **["type": "image"]**]]]
The animal in the image is a dog.

> /image /tmp/img2.jpeg


> describe the second image
[["content": [["text": "You are a helpful assistant who answers questions in English.", "type": "text"]], "role": "system"], ["content": [["text": "what animal is in the image?", "type": "text"], **["type": "image"]**], "role": "user"], ["content": [["type": "text", "text": "The animal in the image is a dog."]], "role": "assistant"], ["content": [["type": "text", "text": "describe the second image"], **["type": "image"]**], "role": "user"]]
The image shows a dog wearing a Santa hat.

Ideally this would present the second image for the second image marker. As it is today it will combine both images and inject them for the first marker.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the UserInputProcessor conversion and the LMInput definition shown in the issue, then trace the model entry point that consumes LMInput. Update the representation and consuming model path so multiple media items remain associated with their respective message markers. Done means a multi-turn chat can present the second image at the second image marker instead of combining both images at the first.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.