bigscience-workshop / bigscience-workshop/Megatron-DeepSpeed
Need model size dumped at init
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 226
- PR merge metrics
- No merged PRs in 30d
Description
We need to have a diagnostic model size dumped during the framework init. We currently get a report per rank and not the total.
```
> number of parameters on (tensor, pipeline) model parallel rank (0, 1): 1745293312
> number of parameters on (tensor, pipeline) model parallel rank (2, 1): 1745293312
> number of parameters on (tensor, pipeline) model parallel rank (3, 0): 1986465792
> number of parameters on (tensor, pipeline) model parallel rank (3, 7): 1986498560
```
Later on ZeRO engine does dump the right thing amongst multiple other numbers and repeated on each rank
```
[2021-10-02 16:08:53,028] [INFO] [engine.py:134:__init__] RANK=0 STAGE=0 LAYERS=7 [0, 7) STAGE_PARAMS=1986465792 (1986.466M) TOTAL_PARAMS=57778896896 (57778.897M) UNIQUE_PARAMS=56814206976 (56814.207M)
```
But ideally we just want a print like:
```
Model size: 57B (57778896896 params)
```
Just on rank 0.
Thanks.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.