michaelfeil / michaelfeil/infinity
How to limit memory usage?
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 206
- PR merge metrics
- No merged PRs in 30d
Description
When I run
```bash
port=3000
model1=Salesforce/SFR-Embedding-Code-2B_R
volume=$PWD/data
docker run -it --gpus device=0 \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest \
v2 \
--model-id $model1 \
--port $port \
--model-warmup \
--batch-size 4
```
I got this warning
```
accelerate.utils.modeling INFO: We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).
```
How can I set `max_memory` for e.g. using only 30% of GPU 0?
Contributor guide
No contributing guide indexed for this repository
Research direction
No file, test, or entry point is named. Start by tracing the `v2` Docker command's model-loading options and how the serving engine passes memory settings to Accelerate; done means documenting and verifying how to limit GPU 0 to 30%.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100