michaelfeil / michaelfeil/infinity

Latest version of transformers breaks Infinity Server

Open
#590 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.9k
Forks
206
PR merge metrics
No merged PRs in 30d

Description

### System Info

```
INFO 2025-05-20 20:03:58,938 datasets INFO: PyTorch version config.py:54
2.6.0 available.
Traceback (most recent call last):
File "/app/.venv/bin/infinity_emb", line 4, in
from infinity_emb.cli import cli
File "/app/.venv/lib/python3.12/site-packages/infinity_emb/__init__.py", line 27, in
from infinity_emb.engine import AsyncEmbeddingEngine, AsyncEngineArray # noqa: E402
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/infinity_emb/engine.py", line 11, in
from infinity_emb.inference import (
File "/app/.venv/lib/python3.12/site-packages/infinity_emb/inference/__init__.py", line 4, in
from infinity_emb.inference.batch_handler import BatchHandler
File "/app/.venv/lib/python3.12/site-packages/infinity_emb/inference/batch_handler.py", line 39, in
from infinity_emb.transformer.utils import get_lengths_with_tokenize
File "/app/.venv/lib/python3.12/site-packages/infinity_emb/transformer/utils.py", line 9, in
from infinity_emb.transformer.classifier.torch import SentenceClassifier
File "/app/.venv/lib/python3.12/site-packages/infinity_emb/transformer/classifier/torch.py", line 8, in
from infinity_emb.transformer.acceleration import (
File "/app/.venv/lib/python3.12/site-packages/infinity_emb/transformer/acceleration.py", line 11, in
from optimum.bettertransformer import ( # type: ignore[import-untyped]
File "/app/.venv/lib/python3.12/site-packages/optimum/bettertransformer/__init__.py", line 20, in
raise RuntimeError(
RuntimeError: BetterTransformer requires transformers<4.49 but found 4.50.0. `optimum.bettertransformer` is deprecated and will be removed in optimum v2.0.

```

We might need to pin the transformers version or maybe even deprecate the better transformer version

### Information

- [x] Docker + cli
- [ ] pip + cli
- [ ] pip + usage of Python interface

### Tasks

- [x] An officially supported CLI command
- [ ] My own modifications

### Reproduction

uv run infinity_emb v2 --model-id `any-model` --port 8000 --url-prefix /v1

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the reported Docker CLI reproduction: `uv run infinity_emb v2 --model-id any-model --port 8000 --url-prefix /v1`, then inspect the dependency configuration and the `optimum.bettertransformer` import path shown in the traceback. Done means the supported CLI starts without the transformers/BetterTransformer incompatibility, with the chosen dependency or deprecation behavior covered by the project’s relevant checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
backend, cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.