Lightning-AI / Lightning-AI/litgpt

How to set max_iters

Open
#1,445 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13.7k
Forks
1.5k
Avg merge
15h 37m
Merged PRs (30d)
1

Description

I am trying to pretrain a model on book corpus, and i was wondering how do i make sure the training runs for 10,000 iterations?

There a max_iters in eval, but not in train.

I am just using the documentation from the home page.

litgpt pretrain \ --model_name pythia-160m \ --tokenizer_dir checkpoints/EleutherAI/pythia-160m \ --data TextFiles \ --data.train_data_path "custom_texts/" \ --train.max_tokens 10_000_000 \ --out_dir out/custom-model

How do i pass the max iters value?
Thanks

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the homepage pretraining documentation and the litgpt pretrain entry point shown in the issue, then inspect how train.max_tokens relates to iteration limits. Document the supported way to run for 10,000 iterations, or clarify that configuration in the example and verify the command matches the documented behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.