michaelfeil / michaelfeil/infinity

How to limit memory usage?

Open
#573 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.9k
Forks
206
PR merge metrics
No merged PRs in 30d

Description

When I run

```bash
port=3000
model1=Salesforce/SFR-Embedding-Code-2B_R
volume=$PWD/data
docker run -it --gpus device=0 \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest \
v2 \
--model-id $model1 \
--port $port \
--model-warmup \
--batch-size 4
```

I got this warning
```
accelerate.utils.modeling INFO: We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).
```

How can I set `max_memory` for e.g. using only 30% of GPU 0?

Contributor guide

No contributing guide indexed for this repository

Research direction

No file, test, or entry point is named. Start by tracing the `v2` Docker command's model-loading options and how the serving engine passes memory settings to Accelerate; done means documenting and verifying how to limit GPU 0 to 30%.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.