Lightning-AI / Lightning-AI/litgpt
Defaults fail on small block size for some models
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 1.5k
- Avg merge
- 15h 37m
- Merged PRs (30d)
- 1
Description
I think that all models should all work out of the box with the defaults. Maybe we need to do some automatic clipping on the inputs and raise a warning if the models have a small block size.
Steps to reproduce
```bash
litgpt download --repo_id EleutherAI/pythia-14m
litgpt finetune lora --checkpoint_dir checkpoints/EleutherAI/pythia-14m
```
```
'lora_mlp': False,
'lora_projection': False,
'lora_query': True,
'lora_r': 8,
'lora_value': True,
'out_dir': PosixPath('out/finetune/lora'),
'precision': None,
'quantize': None,
'seed': 1337,
'train': TrainArgs(save_interval=1000,
log_interval=1,
global_batch_size=128,
micro_batch_size=4,
lr_warmup_steps=100,
epochs=5,
max_tokens=None,
max_steps=None,
max_seq_length=None,
tie_embeddings=None,
learning_rate=0.0003,
weight_decay=0.02,
beta1=0.9,
beta2=0.95,
max_norm=None,
min_lr=6e-05)}
Using bfloat16 Automatic Mixed Precision (AMP)
Seed set to 1337
Number of trainable parameters: 24,576
Number of non-trainable parameters: 14,067,712
Traceback (most recent call last):
File "/home/zeus/miniconda3/envs/cloudspace/bin/litgpt", line 8, in
sys.exit(main())
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/litgpt/__main__.py", line 129, in main
fn(**kwargs)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/litgpt/finetune/lora.py", line 140, in setup
fabric.launch(main, devices, seed, config, data, checkpoint_dir, out_dir, train, eval)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/fabric/fabric.py", line 866, in launch
return self._wrap_and_launch(function, self, *args, **kwargs)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/fabric/fabric.py", line 952, in _wrap_and_launch
return to_run(*args, **kwargs)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/fabric/fabric.py", line 957, in _wrap_with_setup
return to_run(*args, **kwargs)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/litgpt/finetune/lora.py", line 193, in main
fit(
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/litgpt/finetune/lora.py", line 239, in fit
model.max_seq_length = min(longest_seq_length, train.max_seq_length or float("inf"))
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/fabric/wrappers.py", line 276, in __setattr__
setattr(original_module, name, value)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in __setattr__
super().__setattr__(name, value)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/litgpt/model.py", line 47, in max_seq_length
raise ValueError(f"Cannot attend to {value}, block size is only {self.config.block_size}")
ValueError: Cannot attend to 1035, block size is only 512
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with the listed litgpt download and finetune commands, then inspect litgpt/finetune/lora.py around fit and litgpt/model.py around the max_seq_length setter. Done means the default finetune path handles models with smaller block sizes without the shown ValueError and provides the requested warning or clipping behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100