deepspeedai / deepspeedai/DeepSpeedExamples
CPU OOM when inferencing Llama3-70B-Chinese-Chat
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Description
Code: text-generation demo
Command:
deepspeed --num_gpus 2 inference-test.py --dtype float16 --batch_size 4 --max_new_tokens 200 --model ../Llama3-70B-Chinese-Chat
Hardware: two A100 80GB GPUs, CPU 250GB
Problem: When using Deepspeed to load the float16 model, it consumes too much CPU memory, and 250GB of memory cannot load the 70B model. When I use the built-in model of Transformers for inference, Model=AutoModelForCausalLM. from_pretrained (model_id, torch dtype=torch. float16, device_map="auto"), can perform inference without occupying CPU memory.
How to reduce CPU memory usage？
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the command from the issue against inference-test.py with the stated model, dtype, batch size, GPUs, and hardware, then inspect how the example loads the model. Compare its CPU memory use with the Transformers from_pretrained approach described in the report; done means the 70B model can load without exhausting 250GB of CPU memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100