Lightning-AI / Lightning-AI/litgpt
Exclude finetuning datasets from the `pretrain.py` arguments
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 1.5k
- Avg merge
- 15h 37m
- Merged PRs (30d)
- 1
Description
The `pretrain.py` script lists the Alpaca dataset and all other finetuning datasets, but I don't think they are supported for finetuning.
E.g.,
```bash
python litgpt/pretrain.py \
--data litgpt.data.Alpaca2k \
--model_name tiny-llama-1.1b \
--tokenizer_dir checkpoints/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T/
```
```
File "litgpt/pretrain.py", line 381, in
CLI(setup)
File "/teamspace/studios/this_studio/lit-gpt/litgpt/utils.py", line 399, in CLI
return CLI(*args, **kwargs)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.8/site-packages/jsonargparse/_cli.py", line 96, in CLI
return _run_component(components, cfg_init)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.8/site-packages/jsonargparse/_cli.py", line 193, in _run_component
return component(**cfg)
File "litgpt/pretrain.py", line 90, in setup
main(fabric, devices, seed, resume, config, data, out_dir, tokenizer_dir, tokenizer, train, eval)
File "litgpt/pretrain.py", line 155, in main
fit(fabric, devices, state, train_dataloader, val_dataloader, out_dir, tokenizer_dir, train, eval)
File "litgpt/pretrain.py", line 175, in fit
validate(fabric, model, val_dataloader, max_iters=2) # sanity check
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "litgpt/pretrain.py", line 303, in validate
input_ids = batch[:, 0 : model.max_seq_length].contiguous().long()
TypeError: unhashable type: 'slice'
```
I think we should exclude those finetuning datasets from the pretraining args?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in litgpt/pretrain.py at the data argument definitions and compare the listed datasets with the pretraining data interface. Run the example pretrain command to reproduce the reported validation error; done means finetuning-only datasets are no longer offered as pretraining arguments and supported pretraining data still works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100