partial fc training for about 100k iterations, it cause deadlock.

I got this error after about 100k iterations:
```
W horovod/common/stall_inspector.cc:105] One or more tensors were submitted to be reduced, gathered or broadcasted by subset of ranks and are waiting for remainder of ranks for more than 60 seconds. This may indicate that different ranks are trying to submit different tensors or that only subset of ranks is submitting tensors, which will cause deadlock.
Tue Jun 15 02:26:23 2021[0]:Stalled ranks:
```
Every time I train, I will get a deadlock in 100,000 iterations. how to fix it?

Start with the training configuration and distributed execution path that uses Horovod, then reproduce the stall near 100,000 iterations while collecting the stalled-rank output. The issue names no source files or tests; done means identifying why ranks submit different collectives and confirming a long training run no longer deadlocks.

Written by the indexing model from the issue text.