[OOM] Stage 3

When I run step three I get an out of memory error. the program error location is 1072 line in /usr/local/lib/python3.10/dist-packages/deepspeed-0.10.1+cbf2f61a-py3.10.egg/deepspeed/runtime/engine.py （if not self.dont_change_device:
self.module.to(self.device)）
The program completely copied the model onto the gpu causing a memory overflow.What's the solution
I used the A100 machine and used offload and offload_reference_model when I resubmitted the program.
and I used 32 machines and the gpu was 80 gigabytes of memory.and zero3 strategy. I find that the model is whole model in “self.module.to(self.device)”. so out of memory error occurred.What I'm thinking about is how do I assign the model to different Gpus before that .look forward to your reply

Start with deepspeed/runtime/engine.py at line 1072 and inspect the Stage 3 path around self.module.to(self.device), including the offload and offload_reference_model settings described in the report. Reproduce the out-of-memory behavior on the stated A100 and ZeRO-3 setup, then establish a documented fix or confirmed explanation for why the model is copied to one GPU.