Lightning-AI / Lightning-AI/pytorch-lightning
TQDMProgressBar bug (?) causes DDP to hang
@deependujha is already working on this.
Since Oct 6, 2025.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
https://github.com/Lightning-AI/pytorch-lightning/blob/918a1a62635e41b72131fc153920594f0f64cc5b/src/lightning/pytorch/callbacks/progress/tqdm_progress.py#L285-L286
Is this^ a safe implementation for DDP?
Based on my understanding, this will lead to the following stack trace, ultimately leading to a sync operation on `ResultMetrics` if certain conditions are met. However, `self.train_progress_bar.disable` is only `False` (i.e. enabled) on the rank-zero process, so the metrics computations will only succeed if there happens to be another codepath on non-rank-zero devices that's requesting `self._logger_connector.metrics` at the same time.
Stack trace
https://github.com/Lightning-AI/pytorch-lightning/blob/918a1a62635e41b72131fc153920594f0f64cc5b/src/lightning/pytorch/callbacks/progress/progress_bar.py#L180-L201
https://github.com/Lightning-AI/pytorch-lightning/blob/918a1a62635e41b72131fc153920594f0f64cc5b/src/lightning/pytorch/trainer/trainer.py#L1675-L1683
https://github.com/Lightning-AI/pytorch-lightning/blob/918a1a62635e41b72131fc153920594f0f64cc5b/src/lightning/pytorch/trainer/connectors/logger_connector/logger_connector.py#L253-L258
https://github.com/Lightning-AI/pytorch-lightning/blob/918a1a62635e41b72131fc153920594f0f64cc5b/src/lightning/pytorch/trainer/connectors/logger_connector/logger_connector.py#L232-L237
https://github.com/Lightning-AI/pytorch-lightning/blob/918a1a62635e41b72131fc153920594f0f64cc5b/src/lightning/pytorch/trainer/connectors/logger_connector/result.py#L471-L476
https://github.com/Lightning-AI/pytorch-lightning/blob/918a1a62635e41b72131fc153920594f0f64cc5b/src/lightning/pytorch/trainer/connectors/logger_connector/result.py#L425-L440
I ran into this when debugging an issue in my training script where DDP was hanging at the end of the first epoch. Changing the original code chunk as such resolved the deadlock for me.
```
metrics = self.get_metrics(trainer, pl_module)
if not self.train_progress_bar.disable:
self.train_progress_bar.set_postfix(metrics)
```
To provide a little more detail, I have a `CustomCallback` that calls `trainer.strategy.barrier()` during `on_train_epoch_end`. At the end of the first training epoch, the rank-zero process hangs on `TQDMProgressBar.on_train_epoch_end` and the non-rank-zero processes hang on `CustomCallback.on_train_epoch_end`.
cc @justusschock @lantiga
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.