Lightning-AI / Lightning-AI/litgpt

No checkpoints saved during pretraining in Google Colab despite log.log_model and save_interval set.

Open
#2,223 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
13.7k
Forks
1.5k
Avg merge
15h 37m
Merged PRs (30d)
1

Description

### Bug description

Hello,

I've been troubleshooting this issue in a Google Colab environment and wanted to share my findings, as they strongly align with the observations in this thread.

**Troubleshooting Steps & Parameters Used:**
- Initial setup with `bf16-mixed` precision (suitable for T4 GPU).
- Adjusted `train.max_steps` and `train.save_interval` (e.g., both set to 5 for a short run).
- Ensured `log.log_model` was explicitly set to `true` (`--log.log_model true`).
- Explicitly set `log.checkpoint_name` to `'last'` (`--log.checkpoint_name 'last'`)
- Corrected CLI command format (`!python -m litgpt pretrain ...`).
- Removed `train.max_tokens` to avoid premature termination, ensuring `max_steps` was respected.
- Confirmed `litgpt[all]` and other dependencies were installed.

**Observed Outcome:**
Despite all these modifications and successful execution of the `pretrain` command (for short durations, confirming a save should have occurred), no model checkpoints (`.ckpt`, `.pth`) or `model_config.yaml` files were ever saved in the specified `out_dir` (e.g., `out/custom-model/`). Only TensorBoard log files were consistently generated within `out/custom-model/logs/tensorboard/`.

A `find` command within the output directory confirmed the complete absence of any checkpoint-related files.

This behavior suggests an underlying issue within the `litgpt` framework regarding checkpoint saving, at least in the Google Colab environment, and is consistent with what others have reported here. My testing indicates that even with explicit `log.log_model` and `train.save_interval` settings, no checkpoints are being produced.

I hope this detailed context helps in diagnosing the root cause. Please let me know if there are any further tests I can perform. Given these findings, I believe this issue remains relevant.

Thank you.

### Reproduced in studio

_No response_

### What operating system are you using?

Windows

### LitGPT Version

I have attached the Colab notebook I used for these experiments so you can directly reproduce the behavior. If needed, I can also provide a minimal version of the notebook with fewer dependencies and a smaller dataset to make debugging easier. Please let me know if there are any specific logging flags, environment variables, or debug modes you would like me to enable in a follow-up run so I can share more detailed logs or traces.

[litegpt_1402 (1).ipynb](https://github.com/user-attachments/files/26277932/litegpt_1402.1.ipynb)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the attached litegpt_1402.1.ipynb notebook and the `litgpt pretrain` command with `train.save_interval`, `log.log_model`, and `log.checkpoint_name` set as described. Inspect `out/custom-model/` after the short run and compare its contents with `out/custom-model/logs/tensorboard/`; done means the expected checkpoint and `model_config.yaml` files are saved.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.