Lightning-AI / Lightning-AI/pytorch-lightning
Progress bar is broken when loading trainer state from checkpoint
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
I am using lightning in conjunction with the mosaicML streaming library, which allows for stateful dataloaders for resumption of mid-epoch training. I am therefor passing in train/validation dataloaders manually to the trainer, as opposed to a datamodule. That said, as I am also looking to resume with optimizer state etc., I also pass in the checkpoint. Therefor my training is run as:
```
trainer.fit(
model=lightning_model,
train_dataloaders=train_dataloader,
val_dataloaders=validation_dataloader,
ckpt_path=args.ckpt
)
```
Note that at this stage, if resuming, I have already loaded my dataloader and updated with their state dict.
I have confirmed that the dataloader is still returning len(dataloader) correctly, indicating exactly how many steps are in the epoch.
But, when calling with resume logic, for example resuming from step n. 25 I will see the following in progress bar:
`25/?`
So, it seems that the trainer has (correctly) deduced that the checkpoint is resuming from a global step of 25, but is not calling len(dataloader) anymore to verify how many steps remain.
### What version are you seeing the problem on?
v2.5
### How to reproduce the bug
```python
```
### Error messages and logs
_No response_
### Environment
_No response_
### More info
_No response_
cc @lantiga
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the trainer.fit call using manually supplied train_dataloaders, val_dataloaders, and ckpt_path, and reproduce resumption from a nonzero global step. Trace checkpoint resume and progress-bar handling to find why the dataloader length is not shown; done means the resumed progress bar displays the epoch total instead of ?.】【。} jingï 天天彩票与你同行 三分彩 qq的天天中彩票 rogue? Wait malformed? Need ensure JSON valid and no weird. The final currently has Armenian? Actually generated trailing
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100