is FastChat suppport Embedding model such as gte-base?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
I tried to use model_work to deploy an Embedding model gte-base
CUDA_VISIBLE_DEVICES=1 python -m fastchat.serve.model_worker --model-name 'gte-base' --model-path /model/gte-base --debug True --worker-address http://0.0.0.0:21009/ --port 21009 --host 0.0.0.0 --controller-address http://0.0.0.0:21001/
and I got the error which is
Some weights of BertLMHeadModel were not initialized from the model checkpoint at /model/gte-base and are newly initialized: ['cls.predictions.bias', 'cls.predictions.decoder.bias]
I use the transformers and pytorch with the code on the repo model card in the same environment and it works normally
import torch.nn.functional as F
from torch import Tensor
from transformers import AutoTokenizer, AutoModel
def average_pool(last_hidden_states: Tensor,
attention_mask: Tensor) -> Tensor:
last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]
input_texts = [
"what is the capital of China?",
"how to implement quick sort in python?",
"Beijing",
"sorting algorithms"
]
tokenizer = AutoTokenizer.from_pretrained("/model/gte-base")
model = AutoModel.from_pretrained("/model/gte-base")
# Tokenize the input texts
batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt')
outputs = model(**batch_dict)
embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
# (Optionally) normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:1] @ embeddings[1:].T) * 100
print(scores.tolist())
So I would like to ask if FastChat support this embedding or model_worker is the correct to deploy embedding model? Thanks in advance
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the model_worker entry point and reproduce the provided gte-base command, then compare its loading behavior with the repository model-card example using AutoTokenizer and AutoModel. Done means determining whether embedding models are supported by model_worker and, if not, defining the required support or documenting the correct deployment path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- ai, backend, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100