michaelfeil / michaelfeil/infinity
Write a custom flash-attention function for the deberta model.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 206
- PR merge metrics
- No merged PRs in 30d
Description
### Model description
I used `michaelf34/infinity:0.0.55` to deploy mixed_bread_large reranker.
The container is up and I am well capable of pinging the model using python requests, but it is a bit slow (100 requests taking 8 seconds, compared to TEI with BGE that take 0.8s for 100 requests, knowing that BGE-large and Mixed_bread _large have the same size of 335M parameters.
What is the best way to optimize the deployment and inference?
### Open source status
- [X] The model implementation is available on transformers
- [X] The model weights are available on huggingface-hub
- [x] I verified that the model is currently not running in the lastest version `pip install infinity_emb[all] --upgrade`
### Provide useful links for the implementation
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the deployment details in the issue: infinity 0.0.55 serving mixed_bread_large through Python requests, and the reported comparison with TEI and BGE. Investigate the DeBERTa model path and determine how a custom flash-attention implementation would be integrated; completion would require demonstrating improved inference throughput.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100