facebookresearch / facebookresearch/matrix

Support prompt embeds

Open
#62 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
291
Forks
44
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe:**
Currently matrix inference only support passing text instead of passing prompt embeds. However, this is supported by vllm https://docs.vllm.ai/en/stable/features/prompt_embeds.html

**Describe the solution you would like:**
Allowing users to pass prompt_embeds instead of prompts as the payload

**Describe the alternatives you have considered:**
vllm engines deployed locally can be used, but they do not scale.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.