tiny llama 1.1B test shows extra memory consumption after loading the weights and before compilation
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
In this new unit test - test_trtllm_bench_backend_comparison , when commenting out the num_hidden_layers (set to 10 to shorten the test duration), I see that the consumed memory is
Found free_mem_pre_mb: 76943
Total GPU memory: 81007MB
the model params should consume 2.2GB but we see 4GB are consumed.
pytest -s -v tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py::test_trtllm_bench_backend_comparison
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.