michaelfeil / michaelfeil/infinity

Write a custom flash-attention function for the deberta model.

Open
#359 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.9k
Forks
206
PR merge metrics
No merged PRs in 30d

Description

### Model description

I used `michaelf34/infinity:0.0.55` to deploy mixed_bread_large reranker.

The container is up and I am well capable of pinging the model using python requests, but it is a bit slow (100 requests taking 8 seconds, compared to TEI with BGE that take 0.8s for 100 requests, knowing that BGE-large and Mixed_bread _large have the same size of 335M parameters.

What is the best way to optimize the deployment and inference?

### Open source status

- [X] The model implementation is available on transformers
- [X] The model weights are available on huggingface-hub
- [x] I verified that the model is currently not running in the lastest version `pip install infinity_emb[all] --upgrade`

### Provide useful links for the implementation

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the deployment details in the issue: infinity 0.0.55 serving mixed_bread_large through Python requests, and the reported comparison with TEI and BGE. Investigate the DeBERTa model path and determine how a custom flash-attention implementation would be integrated; completion would require demonstrating improved inference throughput.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.