Lightning-AI / Lightning-AI/litgpt

Add tutorial for custom checkpoints

Open
#591 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

model-weights
Dominant language
Python
Stars
13.7k
Forks
1.5k
Avg merge
15h 37m
Merged PRs (30d)
1

Description

If I were to pretrain a model with my own data with something like Phi 1.5, how do I go about using the model? I have only a `last.ckpt` file containing all the weights and optimizers. The pretraining guide doesn't say how this file should be converted to something lit-gpt can use. I've also created the other config files, which the tutorial also lacks for a custom pretrained model. For instance, using my own tokenizer instead of the open source pretrained models'.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the pretraining guide and the custom `last.ckpt` scenario described in the issue. Document how a checkpoint containing weights and optimizers can be converted for litGPT, and include the missing config-file and custom-tokenizer steps. Done means a user with their own pretrained model can follow the tutorial through loading and using it.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.