michaelfeil / michaelfeil/infinity

Jina reranker(turbo/tiny) being classified as embedding models

Open
#325 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.9k
Forks
206
PR merge metrics
No merged PRs in 30d

Description

### System Info

System Info:

AWS EC2 G4dn

Amazon Linux

Model:jinaai/jina-reranker-v1-tiny-en or jinaai/jina-reranker-v1-turbo-en

Hardware: Nvidia-smi

Using latest docker version

Command:

port=7997
rerank_model=jinaai/jina-reranker-v1-tiny-en
volume=$PWD/data

sudo docker run -it --gpus all \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest \
v2 \
--batch-size 256 \
--model-id $rerank_model \
--port $port

### Information

- [X] Docker
- [ ] The CLI directly via pip

### Tasks

- [x] An officially supported command
- [ ] My own modifications

### Reproduction

Run the following command,

port=7997
rerank_model=jinaai/jina-reranker-v1-tiny-en
volume=$PWD/data

sudo docker run -it --gpus all \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest \
v2 \
--batch-size 256 \
--model-id $rerank_model \
--port $port

Then attempt to use the /rerank endpoint with a simple body

{
"query": "test",
"documents": [
"test"
],
"return_documents": false,
"model": "jinaai/jina-reranker-v1-tiny-en"
}

and you will get the following error

"error": {
"message": "ModelNotDeployedError: model=`jinaai/jina-reranker-v1-tiny-en` does not support `rerank`. Reason: the loaded moded cannot fullyfill `rerank`.options are {'embed'}.",
"type": null,
"param": null,
"code": 400
}
}.

I've tested this with other inference servers like Text embedding inference and the same error occurs, however, it does not occur with the standard transformer library.

### Expected behavior

Should be able to rerank with these models.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Docker command and the /rerank request with either listed Jina reranker model. Trace how the loaded model is classified and how supported operations are reported; done means the model is recognized as supporting rerank and the request succeeds without the embed-only error.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.