kubeflow / kubeflow/mpi-operator

Launcher should not collect all the workers' logs

Open
#226 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
535
Forks
238
Avg merge
2d 3h
Merged PRs (30d)
10

Description

I do not know why the launcher will collect all the logs. Just copy the **rank0's** looks more useful. Because i want to print some information in woker1 for debug. But it will be robbed by the laucher.
```
# Horovod: print output only on first rank.
if hvd.rank() == 0:
....
print('\nTest set: Average loss: {:.4f}, Accuracy: {:.2f}%\n'.format(
test_loss, 100. * test_accuracy))
# Horovod: print test only on other rank.
if hvd.rank() != 0:
print('\ntest!!!\n')
```
```
$ kubectl logs pytorch-mnist-tb-launcher-cnk95
....
Train Epoch: 6 [29440/30000 (98%)] Loss: 0.211814
Train Epoch: 6 [29440/30000 (98%)] Loss: 0.098771

test!!!

```
It make me confused

Contributor guide

Open the contributing guide

Research direction

The issue names no source files or tests. Start by tracing how the MPI launcher exposes worker output through kubectl logs, then reproduce the rank0 and non-rank0 prints shown in the report; done means the launcher behavior for worker logs is defined and matches the intended output without hiding needed debug information.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, pytorch
Domain
devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.