facebookresearch / facebookresearch/SlowFast
When to use SyncBatchnorm
Open
enhancement
- Dominant language
- Python
- Stars
- 7.4k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
Hi, thanks for your great work again!
I notice that `DistributedDataparallel` is used in your codebase for parallelism, but `SyncBatchnorm` is not used for synchronizing BN across different GPUs. From my experience, `SyncBatchnorm` sometimes gives better results (at least for training), though a little bit slower. May I ask when `SynchBatchnorm` should (or should not) be used if trained with `DistributedDataparallel`?
Contributor guide
Assessment
This issue has not been assessed yet.