beehive-lab / beehive-lab/GPULlama3.java

Improve error handling on out of memory exception to provide relevant solution for the llama

Open
#31 1 comment 0 reactions 4 assignees Claimed by @stratika View on GitHub
documentation enhancement
Dominant language
Java
Stars
274
Forks
37
Avg merge
5d 1h
Merged PRs (30d)
2

Description

Currently, if one tries to run a model that requires more than the default memory allocation (e.,g, 7GB), GPULlama3.java throws a tornado out of memory exception.

The error message refers to to a solution to increase GPU heap size through a Tornado flag, but the llama repo expects wrapper calls as `--gpu-memory XGB`

Solution:

Lets try to catch the exception in TornadoMasterPlan class and throw the appripate message to increase heap size that is relevant to the llama implementation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.