Questions on scaling the learning rate

Hi, thanks for the repo! In the provided configs, lr=0.03 is for 256 batch size on 8-gpu machine, and lr=0.015 for 128 batch size on 4-gpu machine. This makes sense to me if we only look at the **total batch size** to apply the linear rule (from "ImageNet in 1hr" paper). However, I remember that DDP averages the gradients from all ranks, which means going from 8 gpus to 4 gpus we are implicitly getting 2x on the learning rate (previously divided by 8, now only divided by 4). Am I understanding it correctly? If so, why are we not taking a further 0.5x on the learning rate (making it 0.0075) to compensate the implicit 2x gain caused by DDP?

If I keep the total batch size as 256 but train on 4-gpu machine, how should I change the learning rate?

Thank you!