Lightning-AI / Lightning-AI/litgpt

Pretraining example from readme fails in Colab

Open
#1,402 6 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

3rd party bug help wanted pre-training
Dominant language
Python
Stars
13.7k
Forks
1.5k
Avg merge
15h 37m
Merged PRs (30d)
1

Description

Running the pretraining example from GitHub fails when run in Google Colab.

```
!pip install 'litgpt[all]'

!mkdir -p custom_texts
!curl https://www.gutenberg.org/cache/epub/24440/pg24440.txt --output custom_texts/book1.txt
!curl https://www.gutenberg.org/cache/epub/26393/pg26393.txt --output custom_texts/book2.txt

# 1) Download a tokenizer
!litgpt download \
--repo_id EleutherAI/pythia-160m \
--tokenizer_only True

# 2) Pretrain the model
!litgpt pretrain \
--model_name pythia-160m \
--tokenizer_dir checkpoints/EleutherAI/pythia-160m \
--data TextFiles \
--data.train_data_path "custom_texts/" \
--train.max_tokens 10_000_000 \
--out_dir out/custom-model

# 3) Chat with the model
!litgpt chat \
--checkpoint_dir out/custom-model/final
```

@awaelchli

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the pretraining example in the README and run its Colab commands, including installation, tokenizer download, pretraining, and chat. Capture the failing step and error, then verify a documented fix by running the complete example successfully in Google Colab.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.