Lightning-AI / Lightning-AI/pytorch-lightning

Progress bar is broken when loading trainer state from checkpoint

Open
#20,603 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug checkpointing progress bar: tqdm ver: 2.5.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

I am using lightning in conjunction with the mosaicML streaming library, which allows for stateful dataloaders for resumption of mid-epoch training. I am therefor passing in train/validation dataloaders manually to the trainer, as opposed to a datamodule. That said, as I am also looking to resume with optimizer state etc., I also pass in the checkpoint. Therefor my training is run as:

```
trainer.fit(
model=lightning_model,
train_dataloaders=train_dataloader,
val_dataloaders=validation_dataloader,
ckpt_path=args.ckpt
)
```

Note that at this stage, if resuming, I have already loaded my dataloader and updated with their state dict.
I have confirmed that the dataloader is still returning len(dataloader) correctly, indicating exactly how many steps are in the epoch.

But, when calling with resume logic, for example resuming from step n. 25 I will see the following in progress bar:
`25/?`
So, it seems that the trainer has (correctly) deduced that the checkpoint is resuming from a global step of 25, but is not calling len(dataloader) anymore to verify how many steps remain.

### What version are you seeing the problem on?

v2.5

### How to reproduce the bug

```python

```

### Error messages and logs

_No response_

### Environment

_No response_

### More info

_No response_

cc @lantiga

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the trainer.fit call using manually supplied train_dataloaders, val_dataloaders, and ckpt_path, and reproduce resumption from a nonzero global step. Trace checkpoint resume and progress-bar handling to find why the dataloader length is not shown; done means the resumed progress bar displays the epoch total instead of ?.】【。} jingï 天天彩票与你同行 三分彩 qq的天天中彩票 rogue? Wait malformed? Need ensure JSON valid and no weird. The final currently has Armenian? Actually generated trailing

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.