deepmodeling / deepmodeling/DeePTB

Stabilizing Training with Small Batch Sizes using Exponential Moving Average (EMA)

Open
#217 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
122
Forks
36
PR merge metrics
No merged PRs in 30d

Description

### Background

Small batch sizes are commonly used in Hamiltonian AI training, as seen in [the universal DeepH](https://doi.org/10.1016/j.scib.2024.06.011). However, this approach can lead to violent updates of model parameters.

Specific Observation (DeepTB on QH9 Dataset with Batch Size=1):
![Image](https://github.com/user-attachments/assets/3ab05e2c-7caf-43b1-b606-e0869685d270)

### Describe the solution you'd like

Address the existing [TODO](https://github.com/deepmodeling/DeePTB/blob/57261bb10e5df0b26b9e0170c1becf31e954cc9f/dptb/nnops/trainer.py#L137) regarding EMA implementation to ensure consistent training behavior across related projects.

### Additional Context

This approach is already integrated into [QHNet](https://github.com/divelab/AIRS/blob/e41a755aba8b74f9f6d4372e0e9d654da38b1343/OpenDFT/QHBench/QH9/main.py#L11C23-L11C47) as a default setting.

Contributor guide

Open the contributing guide

Research direction

Start at the EMA TODO in dptb/nnops/trainer.py around line 137, then compare the linked QHNet implementation in OpenDFT/QHBench/QH9/main.py. Determine the expected EMA behavior for small-batch training and how it should fit the existing trainer flow. Done means the TODO is implemented and training behavior is consistent with the stated EMA approach.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.