[Bug/Model Request]: Is slower than sentence transformer for all-minilm-l6-v2
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.2k
- Forks
- 248
- Avg merge
- 4d 8h
- Merged PRs (30d)
- 4
Description
What happened?
On benchmarking synchronous computation times for generating embeddings for
- Using sentence transformers: ~1300 msgs per sec
from sentence_transformers import SentenceTransformer
model_standard = SentenceTransformer("all-MiniLM-L6-v2")
start_time = time.time()
for i in range(iter_count):
model_standard.encode(random.sample(sentences, 1)[0])
time_standard = time.time() - start_time
print("Standard requires: {}s".format(time_standard))
print("{} processed per sec".format(batch_size*iter_count/time_standard))
VS
- Using FastEmbed (Synchronously): 800 msgs per sec
fast_model = TextEmbedding(model_name="sentence-transformers/all-MiniLM-L6-v2")
start_time = time.time()
for i in range(iter_count):
list(fast_model.embed(random.sample(sentences, 1)[0]))
time_standard = time.time() - start_time
print("Fast requires: {}s".format(time_standard))
print("{} processed per sec".format(batch_size*iter_count/time_standard))
I am using fastembed 0.3.3
pip show fastembed
Name: fastembed
Version: 0.3.3
Summary: Fast, light, accurate library built for retrieval embedding generation
Home-page: https://github.com/qdrant/fastembed
Author: Qdrant Team
Author-email: info@qdrant.tech
License: Apache License
Location: /Users/<>/PycharmProjects/Voyager/venv/lib/python3.9/site-packages
Requires: tqdm, PyStemmer, numpy, mmh3, onnxruntime, pillow, onnx, loguru, tokenizers, huggingface-hub, snowballstemmer, requests
Required-by:
Why is this working so slow wrt original impl.? What can I do to improve performance ?
What Python version are you on? e.g. python --version
3.9.16
Version
0.2.7 (Latest)
What os are you seeing the problem on?
MacOS
Relevant stack traces and/or logs
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the comparison using FastEmbed 0.3.3, Python 3.9.16 on macOS, and the all-MiniLM-L6-v2 model. Start at TextEmbedding and its synchronous embed call, compare it with SentenceTransformer.encode, and verify whether the benchmark uses equivalent inputs and batching. Done means identifying the performance difference and documenting or implementing a justified improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100