llama_7b model OOM issue
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 346
- PR merge metrics
- No merged PRs in 30d
Description
Hi
I duplicate the llama model and rename it into llama_7b, changed the model parameters according to llama_7b specification, looks like this:
skiped the CPU eager mode, only run the cuda model.
it reports the following issue when running with this command:
python userbenchmark/dynamo/dynamobench/torchbench.py -dcuda --float16 -n1 --inductor --performance --inference --filter "llama" --batch_size 1 --in_slen 32 --out_slen 3 --output-dir=torchbench_llama_test_logs
If I want to run this model, how should I fix it? my hardware is A100-40G
thanks
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with userbenchmark/dynamo/dynamobench/torchbench.py and reproduce the reported CUDA command using the llama filter, float16, and batch size 1. Compare the duplicated llama_7b parameters with the available model configuration and inspect the reported A100-40G out-of-memory failure. Done means identifying the cause and documenting a verified configuration or limitation for running this model.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100