michaelfeil / michaelfeil/infinity
Infinity embed crashes too easily
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 206
- PR merge metrics
- No merged PRs in 30d
Description
### System Info
0.0.74
### Information
- [X] Docker + cli
- [ ] pip + cli
- [ ] pip + usage of Python interface
### Tasks
- [X] An officially supported CLI command
- [ ] My own modifications
### Reproduction
docker with command: >
v2
--model-id Alibaba-NLP/gte-large-en-v1.5
--batch-size 8
--url-prefix "/v1"
--port 80
Initially, the GPU memory usage starts at just a few gigabytes. However, after running hundreds of calls, the memory consumption gradually increases to over 40GB, eventually resulting in an OOM (Out of Memory) error.
The API should be robust enough to handle heavy usage without crashing or becoming unresponsive, as such issues hinder its usability and reliability. A potential solution could involve implementing a restriction, such as automatically truncating documents that exceed a specified size.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the memory growth through the Docker CLI using the v2 command, Alibaba-NLP/gte-large-en-v1.5, batch size 8, and repeated API calls. Trace the serving path used by this command and observe GPU memory during sustained requests. Done means heavy usage no longer causes unbounded memory growth, an OOM, or an unresponsive API.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- api, backend, cli, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100