Lightning-AI / Lightning-AI/pytorch-lightning

Progress bar doesn't show up on Kaggle TPU with `num_workers` greater than `0`.

Open
#9,814 33 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

accelerator: tpu bug help wanted
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

## 🐛 Bug

As the issue title says: the progress bar doesn't show up on Kaggle TPU with `num_workers` greater than `0`.

Disclaimer: I haven't tested this program on Google Colab TPU.

### To Reproduce

Set `num_workers` to any number greater than zero up to max CPU cores. On Kaggle, the following code sets it to `4`.

```python
train_dataset_loader = DataLoader(train_dataset,
batch_size=BATCH_SIZE,
shuffle=True,
num_workers=multiprocessing.cpu_count(),
drop_last=True)
```

The training is successful but instead of showing progress bar the following output is shown:

```
2021-10-04 04:35:45.588313: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/callback_hook.py:103: LightningDeprecationWarning: The signature of `Callback.on_train_epoch_end` has changed in v1.3. `outputs` parameter has been removed. Support for the old signature will be removed in v1.5
"The signature of `Callback.on_train_epoch_end` has changed in v1.3."
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/tpu_spawn.py:192: UserWarning: cleaning up tpu spawn environment...
rank_zero_warn("cleaning up tpu spawn environment...")
```

### Expected behavior

The progrès bar must show up.

### Environment

- PyTorch Lightning Version: 1.4.9
- PyTorch Version: 1.8.0a0+6e9f2c8
- Python version: 3.7.10
- OS (e.g., Linux): Linux
- CUDA/cuDNN version: N/A
- GPU models and configuration: N/A
- How you installed PyTorch (`conda`, `pip`, source): pip
- If compiling from source, the output of `torch.__config__.show()`: N/A
- Any other relevant information: TPU on Kaggle

### Additional context

N/A

cc @kaushikb11 @rohitgr7 @tchaton

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the training setup on a Kaggle TPU with a DataLoader using num_workers greater than zero, then compare it with num_workers set to zero. Investigate the TPU worker and progress-bar output paths; done means the progress bar appears during training without breaking the successful run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.