facebookresearch / facebookresearch/matrix

Support image/video in Inference Command

Open
#47 0 comments 0 reactions 1 assignee Claimed by @cfyehtw View on GitHub
enhancement
Dominant language
Python
Stars
291
Forks
44
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe:**

Currently matrix inference only support text.

**Describe the solution you would like:**

1. Find huggingface dataset contains images (eg cais/hle) and videos
2. Convert the row into Chat message
3. Deploy maverick-fp8 on h100 and test it on images, which should work.
4. Explore video input and model, eg gemma

**Additional Context:**
We currently integrated vllm v0.8.3, please see if it support video models.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.