kubeflow / kubeflow/mpi-operator
Launcher should not collect all the workers' logs
- Dominant language
- Go
- Stars
- 535
- Forks
- 238
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 10
Description
I do not know why the launcher will collect all the logs. Just copy the **rank0's** looks more useful. Because i want to print some information in woker1 for debug. But it will be robbed by the laucher.
```
# Horovod: print output only on first rank.
if hvd.rank() == 0:
....
print('\nTest set: Average loss: {:.4f}, Accuracy: {:.2f}%\n'.format(
test_loss, 100. * test_accuracy))
# Horovod: print test only on other rank.
if hvd.rank() != 0:
print('\ntest!!!\n')
```
```
$ kubectl logs pytorch-mnist-tb-launcher-cnk95
....
Train Epoch: 6 [29440/30000 (98%)] Loss: 0.211814
Train Epoch: 6 [29440/30000 (98%)] Loss: 0.098771
test!!!
```
It make me confused
Contributor guide
Research direction
The issue names no source files or tests. Start by tracing how the MPI launcher exposes worker output through kubectl logs, then reproduce the rank0 and non-rank0 prints shown in the report; done means the launcher behavior for worker logs is defined and matches the intended output without hiding needed debug information.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, pytorch
- Domain
- devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100