lm-sys / lm-sys/FastChat

is FastChat suppport Embedding model such as gte-base?

Open
#3,181 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

I tried to use model_work to deploy an Embedding model gte-base

CUDA_VISIBLE_DEVICES=1 python -m fastchat.serve.model_worker --model-name 'gte-base' --model-path /model/gte-base --debug True --worker-address http://0.0.0.0:21009/ --port 21009 --host 0.0.0.0 --controller-address http://0.0.0.0:21001/

and I got the error which is

Some weights of BertLMHeadModel were not initialized from the model checkpoint at /model/gte-base and are newly initialized: ['cls.predictions.bias', 'cls.predictions.decoder.bias]

I use the transformers and pytorch with the code on the repo model card in the same environment and it works normally

import torch.nn.functional as F
from torch import Tensor
from transformers import AutoTokenizer, AutoModel

def average_pool(last_hidden_states: Tensor,
                 attention_mask: Tensor) -> Tensor:
    last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
    return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]

input_texts = [
    "what is the capital of China?",
    "how to implement quick sort in python?",
    "Beijing",
    "sorting algorithms"
]

tokenizer = AutoTokenizer.from_pretrained("/model/gte-base")
model = AutoModel.from_pretrained("/model/gte-base")

# Tokenize the input texts
batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt')

outputs = model(**batch_dict)
embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])

# (Optionally) normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:1] @ embeddings[1:].T) * 100
print(scores.tolist())

So I would like to ask if FastChat support this embedding or model_worker is the correct to deploy embedding model? Thanks in advance

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the model_worker entry point and reproduce the provided gte-base command, then compare its loading behavior with the repository model-card example using AutoTokenizer and AutoModel. Done means determining whether embedding models are supported by model_worker and, if not, defining the required support or documenting the correct deployment path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
ai, backend, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.