huggingface / huggingface/local-gemma

memory preset results in UnboundLocalError

Open
#43 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
383
Forks
31
PR merge metrics
No merged PRs in 30d

Description

local-gemma version 0.2.0, pipx install on Linux, NVidia RTX 3090:

```console
user:~$ local-gemma --model="27b" --preset="memory" "What is the capital of Germany?"
Traceback (most recent call last):
File "/home/user/.local/bin/local-gemma", line 8, in
sys.exit(main())
^^^^^^
File "/home/user/.local/share/pipx/venvs/local-gemma/lib/python3.11/site-packages/local_gemma/cli.py", line 178, in main
if spare_memory / 1e9 > 5:
^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'spare_memory' where it is not associated with a value
user:~$ local-gemma --model="9b" --preset="memory" "What is the capital of Germany?"
Traceback (most recent call last):
File "/home/user/.local/bin/local-gemma", line 8, in
sys.exit(main())
^^^^^^
File "/home/user/.local/share/pipx/venvs/local-gemma/lib/python3.11/site-packages/local_gemma/cli.py", line 178, in main
if spare_memory / 1e9 > 5:
^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'spare_memory' where it is not associated with a value
user:~$ local-gemma --model="2b" --preset="memory" "What is the capital of Germany?"

Loading model with the following characteristics:
- Model name: google/gemma-2-2b-it
- Assistant model name: None
- Device: cuda
- Default data type: torch.bfloat16
- Optimization preset: memory
- Generation arguments: {'do_sample': True, 'temperature': 0.7}
- Base prompt: None

Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00, 1.88it/s]
The 'max_batch_size' argument of HybridCache is deprecated and will be removed in v4.46. Use the more precisely named 'batch_size' argument instead.
The capital of Germany is **Berlin**.

user:~$ nvidia-smi
Wed Nov 27 17:48:18 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01 Driver Version: 535.183.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 3090 On | 00000000:01:00.0 Off | N/A |
| 0% 39C P8 29W / 220W | 10MiB / 24576MiB | 1% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 1380 G /usr/lib/xorg/Xorg 4MiB |
+---------------------------------------------------------------------------------------+

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in local_gemma/cli.py around line 178 and reproduce the failure with the reported --model=27b or --model=9b and --preset=memory commands. Compare those runs with the working 2b command; done means the larger models no longer raise UnboundLocalError while the memory preset still loads successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.