michaelfeil / michaelfeil/infinity

Infinity embed crashes too easily

Open
#517 11 comments 5 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.9k
Forks
206
PR merge metrics
No merged PRs in 30d

Description

### System Info

0.0.74

### Information

- [X] Docker + cli
- [ ] pip + cli
- [ ] pip + usage of Python interface

### Tasks

- [X] An officially supported CLI command
- [ ] My own modifications

### Reproduction

docker with command: >
v2
--model-id Alibaba-NLP/gte-large-en-v1.5
--batch-size 8
--url-prefix "/v1"
--port 80

Initially, the GPU memory usage starts at just a few gigabytes. However, after running hundreds of calls, the memory consumption gradually increases to over 40GB, eventually resulting in an OOM (Out of Memory) error.

The API should be robust enough to handle heavy usage without crashing or becoming unresponsive, as such issues hinder its usability and reliability. A potential solution could involve implementing a restriction, such as automatically truncating documents that exceed a specified size.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the memory growth through the Docker CLI using the v2 command, Alibaba-NLP/gte-large-en-v1.5, batch size 8, and repeated API calls. Trace the serving path used by this command and observe GPU memory during sustained requests. Done means heavy usage no longer causes unbounded memory growth, an OOM, or an unresponsive API.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
api, backend, cli, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.