deepmodeling / deepmodeling/unimol_tools

[Code scan] Reload best checkpoint for DDP validation predictions

Open
#37 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34
Forks
9
PR merge metrics
No merged PRs in 30d

Description

This issue is a result of a Codex global repository scan.

## Summary
The non-DDP training path returns validation predictions from the best saved checkpoint, because the final predict call uses load_model=True. The DDP path saves early-stopped checkpoints on rank 0, but its final prediction uses load_model=False, so cross-validation predictions can come from the last epoch rather than the best epoch.

## Code references
https://github.com/deepmodeling/unimol_tools/blob/4596596aa8f73eb462d5cc5a921d79966d0465da/unimol_tools/tasks/trainer.py#L296-L312
https://github.com/deepmodeling/unimol_tools/blob/4596596aa8f73eb462d5cc5a921d79966d0465da/unimol_tools/tasks/trainer.py#L375-L423
https://github.com/deepmodeling/unimol_tools/blob/4596596aa8f73eb462d5cc5a921d79966d0465da/unimol_tools/tasks/trainer.py#L582-L587

## Impact
DDP cross-validation metrics and saved predictions may not correspond to the checkpoint selected by early stopping, making DDP results inconsistent with single-GPU results.

## Suggested fix
Add DDP-aware best-checkpoint loading on every rank before the final prediction, using model.module.load_pretrained_weights or explicit state-dict loading, then gather predictions from the restored model.

Contributor guide

No contributing guide indexed for this repository

Research direction

Read the cited sections of unimol_tools/tasks/trainer.py, especially the non-DDP and DDP prediction paths around lines 296-312, 375-423, and 582-587. Compare how the best checkpoint is selected and loaded, then verify the DDP path restores it on every rank before final prediction. Done means DDP cross-validation predictions use the early-stopped checkpoint consistently with the non-DDP path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.