qdrant / qdrant/fastembed

[Bug/Model Request]: Is slower than sentence transformer for all-minilm-l6-v2

Open
#292 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3.2k
Forks
248
Avg merge
4d 8h
Merged PRs (30d)
4

Description

What happened?

On benchmarking synchronous computation times for generating embeddings for

  1. Using sentence transformers: ~1300 msgs per sec
    from sentence_transformers import SentenceTransformer
    model_standard = SentenceTransformer("all-MiniLM-L6-v2")

    start_time = time.time()
    for i in range(iter_count):
        model_standard.encode(random.sample(sentences, 1)[0])
    time_standard = time.time() - start_time
    print("Standard requires: {}s".format(time_standard))
    print("{} processed per sec".format(batch_size*iter_count/time_standard))

VS

  1. Using FastEmbed (Synchronously): 800 msgs per sec
    fast_model = TextEmbedding(model_name="sentence-transformers/all-MiniLM-L6-v2")
    start_time = time.time()
    for i in range(iter_count):
        list(fast_model.embed(random.sample(sentences, 1)[0]))
    time_standard = time.time() - start_time
    print("Fast requires: {}s".format(time_standard))
    print("{} processed per sec".format(batch_size*iter_count/time_standard))

I am using fastembed 0.3.3

pip show fastembed
Name: fastembed
Version: 0.3.3
Summary: Fast, light, accurate library built for retrieval embedding generation
Home-page: https://github.com/qdrant/fastembed
Author: Qdrant Team
Author-email: info@qdrant.tech
License: Apache License
Location: /Users/<>/PycharmProjects/Voyager/venv/lib/python3.9/site-packages
Requires: tqdm, PyStemmer, numpy, mmh3, onnxruntime, pillow, onnx, loguru, tokenizers, huggingface-hub, snowballstemmer, requests
Required-by: 

Why is this working so slow wrt original impl.? What can I do to improve performance ?

What Python version are you on? e.g. python --version

3.9.16

Version

0.2.7 (Latest)

What os are you seeing the problem on?

MacOS

Relevant stack traces and/or logs

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the comparison using FastEmbed 0.3.3, Python 3.9.16 on macOS, and the all-MiniLM-L6-v2 model. Start at TextEmbedding and its synchronous embed call, compare it with SentenceTransformer.encode, and verify whether the benchmark uses equivalent inputs and batching. Done means identifying the performance difference and documenting or implementing a justified improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.