deepmodeling / deepmodeling/unimol_tools
[Code scan] Reload best checkpoint for DDP validation predictions
- Dominant language
- Python
- Stars
- 34
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
This issue is a result of a Codex global repository scan.
## Summary
The non-DDP training path returns validation predictions from the best saved checkpoint, because the final predict call uses load_model=True. The DDP path saves early-stopped checkpoints on rank 0, but its final prediction uses load_model=False, so cross-validation predictions can come from the last epoch rather than the best epoch.
## Code references
https://github.com/deepmodeling/unimol_tools/blob/4596596aa8f73eb462d5cc5a921d79966d0465da/unimol_tools/tasks/trainer.py#L296-L312
https://github.com/deepmodeling/unimol_tools/blob/4596596aa8f73eb462d5cc5a921d79966d0465da/unimol_tools/tasks/trainer.py#L375-L423
https://github.com/deepmodeling/unimol_tools/blob/4596596aa8f73eb462d5cc5a921d79966d0465da/unimol_tools/tasks/trainer.py#L582-L587
## Impact
DDP cross-validation metrics and saved predictions may not correspond to the checkpoint selected by early stopping, making DDP results inconsistent with single-GPU results.
## Suggested fix
Add DDP-aware best-checkpoint loading on every rank before the final prediction, using model.module.load_pretrained_weights or explicit state-dict loading, then gather predictions from the restored model.
Contributor guide
No contributing guide indexed for this repository
Research direction
Read the cited sections of unimol_tools/tasks/trainer.py, especially the non-DDP and DDP prediction paths around lines 296-312, 375-423, and 582-587. Compare how the best checkpoint is selected and loaded, then verify the DDP path restores it on every rank before final prediction. Done means DDP cross-validation predictions use the early-stopped checkpoint consistently with the non-DDP path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100