🐛 [Bug] Llama-2-7b on a 4090 GPU
Open
Nobody has claimed this yet.
bug
hf
story: LLM & Generative AI
- Dominant language
- Python
- Stars
- 3k
- Forks
- 410
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 78
Description
Bug Description
Currently, Torch-TRT displays the following error when compiling Llama-2-7B in FP16 on a 4090 GPU:
[05/08/2024-20:47:56] [TRT] [E] 1: [defaultAllocator.cpp::allocate::19] Error Code 1: Cuda Runtime (out of memory)
[05/08/2024-20:47:56] [TRT] [W] Requested amount of GPU memory (13476831488 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[05/08/2024-20:47:56] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin (No Myelin Error exists)
The model should successfully compile on a 4090 GPU, given the memory available.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the FP16 Llama-2-7B compilation on a 4090 GPU and capture the exact model, Torch-TRT, PyTorch, and TensorRT configuration. Trace the reported allocator and Myelin out-of-memory failure, then verify that compilation succeeds without exceeding the GPU's available memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- compilers, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100