Lightning-AI / Lightning-AI/litgpt
Pretraining example from readme fails in Colab
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 1.5k
- Avg merge
- 15h 37m
- Merged PRs (30d)
- 1
Description
Running the pretraining example from GitHub fails when run in Google Colab.
```
!pip install 'litgpt[all]'
!mkdir -p custom_texts
!curl https://www.gutenberg.org/cache/epub/24440/pg24440.txt --output custom_texts/book1.txt
!curl https://www.gutenberg.org/cache/epub/26393/pg26393.txt --output custom_texts/book2.txt
# 1) Download a tokenizer
!litgpt download \
--repo_id EleutherAI/pythia-160m \
--tokenizer_only True
# 2) Pretrain the model
!litgpt pretrain \
--model_name pythia-160m \
--tokenizer_dir checkpoints/EleutherAI/pythia-160m \
--data TextFiles \
--data.train_data_path "custom_texts/" \
--train.max_tokens 10_000_000 \
--out_dir out/custom-model
# 3) Chat with the model
!litgpt chat \
--checkpoint_dir out/custom-model/final
```
@awaelchli
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the pretraining example in the README and run its Colab commands, including installation, tokenizer download, pretraining, and chat. Capture the failing step and error, then verify a documented fix by running the complete example successfully in Google Colab.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100