ml-explore / ml-explore/mlx-swift-examples
LMInput restricts model input to a single collection of images and video frames
Nobody has claimed this yet.
- Dominant language
- Swift
- Stars
- 2.7k
- Forks
- 426
- PR merge metrics
- No merged PRs in 30d
Description
See #277 and #276
The UserInput struct can represent a series of messages with media attached to each image:
return UserInput(
chat: [
.system(generate.system),
.user(prompt, images: media.images, videos: media.videos),
],
processing: media.processing
)
This could include back and forth between the user and assistant including adding additional media.
The UserInputProcessor converts this to an LMInput:
public struct LMInput {
public let text: Text
public let image: ProcessedImage?
public let video: ProcessedVideo?
but that only allows for one set of image/video. This should probably have:
public let images: [ProcessedImage]
public let videos: [ProcessedVideo]
though the model would have to be updated to take advantage of that.
Consider this chat:
> /image /tmp/img.jpeg
> what animal is in the image?
[["role": "system", "content": [["text": "You are a helpful assistant who answers questions in English.", "type": "text"]]], ["role": "user", "content": [["text": "what animal is in the image?", "type": "text"], **["type": "image"]**]]]
The animal in the image is a dog.
> /image /tmp/img2.jpeg
> describe the second image
[["content": [["text": "You are a helpful assistant who answers questions in English.", "type": "text"]], "role": "system"], ["content": [["text": "what animal is in the image?", "type": "text"], **["type": "image"]**], "role": "user"], ["content": [["type": "text", "text": "The animal in the image is a dog."]], "role": "assistant"], ["content": [["type": "text", "text": "describe the second image"], **["type": "image"]**], "role": "user"]]
The image shows a dog wearing a Santa hat.
Ideally this would present the second image for the second image marker. As it is today it will combine both images and inject them for the first marker.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the UserInputProcessor conversion and the LMInput definition shown in the issue, then trace the model entry point that consumes LMInput. Update the representation and consuming model path so multiple media items remain associated with their respective message markers. Done means a multi-turn chat can present the second image at the second image marker instead of combining both images at the first.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100