Lightning-AI / Lightning-AI/litgpt
No checkpoints saved during pretraining in Google Colab despite log.log_model and save_interval set.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 1.5k
- Avg merge
- 15h 37m
- Merged PRs (30d)
- 1
Description
### Bug description
Hello,
I've been troubleshooting this issue in a Google Colab environment and wanted to share my findings, as they strongly align with the observations in this thread.
**Troubleshooting Steps & Parameters Used:**
- Initial setup with `bf16-mixed` precision (suitable for T4 GPU).
- Adjusted `train.max_steps` and `train.save_interval` (e.g., both set to 5 for a short run).
- Ensured `log.log_model` was explicitly set to `true` (`--log.log_model true`).
- Explicitly set `log.checkpoint_name` to `'last'` (`--log.checkpoint_name 'last'`)
- Corrected CLI command format (`!python -m litgpt pretrain ...`).
- Removed `train.max_tokens` to avoid premature termination, ensuring `max_steps` was respected.
- Confirmed `litgpt[all]` and other dependencies were installed.
**Observed Outcome:**
Despite all these modifications and successful execution of the `pretrain` command (for short durations, confirming a save should have occurred), no model checkpoints (`.ckpt`, `.pth`) or `model_config.yaml` files were ever saved in the specified `out_dir` (e.g., `out/custom-model/`). Only TensorBoard log files were consistently generated within `out/custom-model/logs/tensorboard/`.
A `find` command within the output directory confirmed the complete absence of any checkpoint-related files.
This behavior suggests an underlying issue within the `litgpt` framework regarding checkpoint saving, at least in the Google Colab environment, and is consistent with what others have reported here. My testing indicates that even with explicit `log.log_model` and `train.save_interval` settings, no checkpoints are being produced.
I hope this detailed context helps in diagnosing the root cause. Please let me know if there are any further tests I can perform. Given these findings, I believe this issue remains relevant.
Thank you.
### Reproduced in studio
_No response_
### What operating system are you using?
Windows
### LitGPT Version
I have attached the Colab notebook I used for these experiments so you can directly reproduce the behavior. If needed, I can also provide a minimal version of the notebook with fewer dependencies and a smaller dataset to make debugging easier. Please let me know if there are any specific logging flags, environment variables, or debug modes you would like me to enable in a follow-up run so I can share more detailed logs or traces.
[litegpt_1402 (1).ipynb](https://github.com/user-attachments/files/26277932/litegpt_1402.1.ipynb)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the attached litegpt_1402.1.ipynb notebook and the `litgpt pretrain` command with `train.save_interval`, `log.log_model`, and `log.checkpoint_name` set as described. Inspect `out/custom-model/` after the short run and compare its contents with `out/custom-model/logs/tensorboard/`; done means the expected checkpoint and `model_config.yaml` files are saved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100