michaelfeil / michaelfeil/infinity
Jina reranker(turbo/tiny) being classified as embedding models
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 206
- PR merge metrics
- No merged PRs in 30d
Description
### System Info
System Info:
AWS EC2 G4dn
Amazon Linux
Model:jinaai/jina-reranker-v1-tiny-en or jinaai/jina-reranker-v1-turbo-en
Hardware: Nvidia-smi
Using latest docker version
Command:
port=7997
rerank_model=jinaai/jina-reranker-v1-tiny-en
volume=$PWD/data
sudo docker run -it --gpus all \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest \
v2 \
--batch-size 256 \
--model-id $rerank_model \
--port $port
### Information
- [X] Docker
- [ ] The CLI directly via pip
### Tasks
- [x] An officially supported command
- [ ] My own modifications
### Reproduction
Run the following command,
port=7997
rerank_model=jinaai/jina-reranker-v1-tiny-en
volume=$PWD/data
sudo docker run -it --gpus all \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest \
v2 \
--batch-size 256 \
--model-id $rerank_model \
--port $port
Then attempt to use the /rerank endpoint with a simple body
{
"query": "test",
"documents": [
"test"
],
"return_documents": false,
"model": "jinaai/jina-reranker-v1-tiny-en"
}
and you will get the following error
"error": {
"message": "ModelNotDeployedError: model=`jinaai/jina-reranker-v1-tiny-en` does not support `rerank`. Reason: the loaded moded cannot fullyfill `rerank`.options are {'embed'}.",
"type": null,
"param": null,
"code": 400
}
}.
I've tested this with other inference servers like Text embedding inference and the same error occurs, however, it does not occur with the standard transformer library.
### Expected behavior
Should be able to rerank with these models.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Docker command and the /rerank request with either listed Jina reranker model. Trace how the loaded model is classified and how supported operations are reported; done means the model is recognized as supporting rerank and the request succeeds without the embed-only error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100