Train partial fc error

I got this error after about 100k iterations:

Fri Oct 23 17:16:00 2020[0]:[2020-10-23 17:16:00.707119: W horovod/common/stall_inspector.cc:105] One or more tensors were submitted to be reduced, gathered or broadcasted by subset of ranks and are waiting for remainder of ranks for more than 60 seconds. This may indicate that different ranks are trying to submit different tensors or that only subset of ranks is submitting tensors, which will cause deadlock.
Fri Oct 23 17:16:00 2020[0]:Stalled ranks:
Fri Oct 23 17:16:00 2020[0]:0: [horovod_allreduce.noname.691941]
Fri Oct 23 17:16:00 2020[0]:1: [horovod_allreduce.noname.691943]
Fri Oct 23 17:16:00 2020[0]:2: [horovod_allreduce.noname.691943]
Fri Oct 23 17:16:00 2020[0]:3: [horovod_allreduce.noname.691943]
Fri Oct 23 17:16:00 2020[0]:4: [horovod_allreduce.noname.691943]
Fri Oct 23 17:16:00 2020[0]:5: [horovod_allreduce.noname.691943]
Fri Oct 23 17:16:00 2020[0]:6: [horovod_allreduce.noname.691943]
Fri Oct 23 17:16:00 2020[0]:7: [horovod_allreduce.noname.691943]

How can i fix this issue? Thank you