Lightning-AI / Lightning-AI/litgpt

Exclude finetuning datasets from the `pretrain.py` arguments

Open
#1,084 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13.7k
Forks
1.5k
Avg merge
15h 37m
Merged PRs (30d)
1

Description

The `pretrain.py` script lists the Alpaca dataset and all other finetuning datasets, but I don't think they are supported for finetuning.

E.g.,

```bash
python litgpt/pretrain.py \
--data litgpt.data.Alpaca2k \
--model_name tiny-llama-1.1b \
--tokenizer_dir checkpoints/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T/
```

```
File "litgpt/pretrain.py", line 381, in
CLI(setup)
File "/teamspace/studios/this_studio/lit-gpt/litgpt/utils.py", line 399, in CLI
return CLI(*args, **kwargs)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.8/site-packages/jsonargparse/_cli.py", line 96, in CLI
return _run_component(components, cfg_init)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.8/site-packages/jsonargparse/_cli.py", line 193, in _run_component
return component(**cfg)
File "litgpt/pretrain.py", line 90, in setup
main(fabric, devices, seed, resume, config, data, out_dir, tokenizer_dir, tokenizer, train, eval)
File "litgpt/pretrain.py", line 155, in main
fit(fabric, devices, state, train_dataloader, val_dataloader, out_dir, tokenizer_dir, train, eval)
File "litgpt/pretrain.py", line 175, in fit
validate(fabric, model, val_dataloader, max_iters=2) # sanity check
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "litgpt/pretrain.py", line 303, in validate
input_ids = batch[:, 0 : model.max_seq_length].contiguous().long()
TypeError: unhashable type: 'slice'
```

I think we should exclude those finetuning datasets from the pretraining args?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in litgpt/pretrain.py at the data argument definitions and compare the listed datasets with the pretraining data interface. Run the example pretrain command to reproduce the reported validation error; done means finetuning-only datasets are no longer offered as pretraining arguments and supported pretraining data still works.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.