GPU Memory is not freed after embedding operations

Open

#514 opened on Jan 14, 2025 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language: Python
Stars: 2.9k
Forks: 206
PR merge metrics: No merged PRs in 30d

Description

### System Info

Infinity docker image : michaelf34/infinity:0.0.74

Docker compose command and deploy parts
```yaml
command: [
"v2",
"--model-id","Alibaba-NLP/gte-multilingual-base",
"--batch-size","4",
"--dtype","float16",
"--device","cuda",
"--engine","torch",
"--port","7997"
]
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities:
- gpu
- utility
- compute
- video
```

### Information

- [X] Docker + cli
- [ ] pip + cli
- [ ] pip + usage of Python interface

### Tasks

- [X] An officially supported CLI command
- [ ] My own modifications

### Reproduction

embedded around 13k documents, waited 6 hours after the embedding batches, the memory GPU memory is still allocated to the process and not freed by embedding memory artifact (not talking about the model itself).

is there a way to free the memory outside of restarting the container ?

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 128020 C /app/.venv/bin/python 1164MiB |
+-----------------------------------------------------------------------------------------+
```

Memory 6 hours after the encoding batch is completed

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1931 C /app/.venv/bin/python 10506MiB |
+-----------------------------------------------------------------------------------------+
```

Contributor guide

No contributing guide indexed for this repository

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Research direction

Start by reproducing the official Docker CLI v2 command with the torch engine and compare nvidia-smi before and after the embedding batch. Trace the embedding batch lifecycle and determine whether memory attributed to completed artifacts can be released without restarting the container, while retaining the model allocation.

Written by the indexing model from the issue text.

Assessment

Tech stack: docker, python
Domain: backend, machine-learning
Issue type: Bug
Difficulty: 4/5
Estimated time: 3-5 days
Activity status: Stale
Clarity: Needs clarification
Newbie friendliness: 25/100