michaelfeil / michaelfeil/infinity

Warmup fails for embedding model

Open
#577 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.9k
Forks
206
PR merge metrics
No merged PRs in 30d

Description

### System Info

Command:
docker compose up

OS Version:
linux, ubuntu

Model:
intfloat/multilingual-e5-large-instruct

docker compose file:

```
services:
infinity:
image: michaelf34/infinity:latest-cpu
command:
- v2
- --engine
- optimum
- --model-id
- intfloat/multilingual-e5-large-instruct
- --served-model-name
- keyword-classifier
- --port
- "5000"
ports:
- "5000:5000"
```

docker image digest: 2a464dcc06e6

### Information

- [x] Docker + cli
- [ ] pip + cli
- [ ] pip + usage of Python interface

### Tasks

- [ ] An officially supported CLI command
- [ ] My own modifications

### Reproduction

Steps to reproduce:
1. Start docker compose with the example file

Expected behaviour:
- Server starts and warmup is executed

Actucal behaviour:
An error occurs: it seems like a too long input is sent to the model for warmup.
It seems like that the input is not truncated. A possible fix could be to add truncation, or to make sure that the input is not too long.

The error:
infinity-1 | INFO 2025-04-29 08:24:13,721 infinity_emb INFO: Getting select_model.py:97
infinity-1 | timings for batch_size=32 and avg tokens per
infinity-1 | sentence=3
infinity-1 | 4.91 ms tokenization
infinity-1 | 97.51 ms inference
infinity-1 | 0.16 ms post-processing
infinity-1 | 102.59 ms total
infinity-1 | embeddings/sec: 311.93
infinity-1 | 2025-04-29 08:24:13.751459785 [E:onnxruntime:, sequential_executor.cc:516 ExecuteKernel] Non-zero status code returned while running Gather node. Name:'/0/auto_model/embeddings/position_embeddings/Gather' Status Message: indices element out of data bounds, idx=514 must be within the inclusive range [-514,513]
infinity-1 | ERROR: Traceback (most recent call last):
infinity-1 | File "/app/.venv/lib/python3.10/site-packages/starlette/routing.py", line 693, in lifespan
infinity-1 | async with self.lifespan_context(app) as maybe_state:
infinity-1 | File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
infinity-1 | return await anext(self.gen)
infinity-1 | File "/app/infinity_emb/infinity_server.py", line 88, in lifespan
infinity-1 | app.engine_array = AsyncEngineArray.from_args(engine_args_list) # type: ignore
infinity-1 | File "/app/infinity_emb/engine.py", line 306, in from_args
infinity-1 | return cls(engines=tuple(engines))
infinity-1 | File "/app/infinity_emb/engine.py", line 71, in from_args
infinity-1 | engine = cls(**engine_args.to_dict(), _show_deprecation_warning=False)
infinity-1 | File "/app/infinity_emb/engine.py", line 56, in __init__
infinity-1 | self._model_replicas, self._min_inference_t, self._max_inference_t = select_model(
infinity-1 | File "/app/infinity_emb/inference/select_model.py", line 99, in select_model
infinity-1 | loaded_engine.warmup(batch_size=engine_args.batch_size, n_tokens=512)
infinity-1 | File "/app/infinity_emb/transformer/abstract.py", line 93, in warmup
infinity-1 | return run_warmup(self, inp)
infinity-1 | File "/app/infinity_emb/transformer/abstract.py", line 233, in run_warmup
infinity-1 | embed = model.encode_core(feat)
infinity-1 | File "/app/infinity_emb/transformer/embedder/optimum.py", line 91, in encode_core
infinity-1 | outputs = self.model(**onnx_input)
infinity-1 | File "/app/.venv/lib/python3.10/site-packages/optimum/modeling_base.py", line 98, in __call__
infinity-1 | return self.forward(*args, **kwargs)
infinity-1 | File "/app/.venv/lib/python3.10/site-packages/optimum/onnxruntime/modeling_ort.py", line 1109, in forward
infinity-1 | onnx_outputs = self.model.run(None, onnx_inputs)
infinity-1 | File "/app/.venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 266, in run
infinity-1 | return self._sess.run(output_names, input_feed, run_options)
infinity-1 | onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running Gather node. Name:'/0/auto_model/embeddings/position_embeddings/Gather' Status Message: indices element out of data bounds, idx=514 must be within the inclusive range [-514,513]
infinity-1 |
infinity-1 | ERROR: Application startup failed. Exiting.
infinity-1 exited with code 3

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with docker compose up using the shown service definition. Start at infinity_emb/transformer/abstract.py, then follow inference/select_model.py and transformer/embedder/optimum.py to trace the warmup input reaching the model. Done means this model starts successfully and warmup completes without the ONNXRuntime position-embedding bounds error.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
backend, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.