Lightning-AI / Lightning-AI/pytorch-lightning
Progress bar doesn't show up on Kaggle TPU with `num_workers` greater than `0`.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🐛 Bug
As the issue title says: the progress bar doesn't show up on Kaggle TPU with `num_workers` greater than `0`.
Disclaimer: I haven't tested this program on Google Colab TPU.
### To Reproduce
Set `num_workers` to any number greater than zero up to max CPU cores. On Kaggle, the following code sets it to `4`.
```python
train_dataset_loader = DataLoader(train_dataset,
batch_size=BATCH_SIZE,
shuffle=True,
num_workers=multiprocessing.cpu_count(),
drop_last=True)
```
The training is successful but instead of showing progress bar the following output is shown:
```
2021-10-04 04:35:45.588313: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/callback_hook.py:103: LightningDeprecationWarning: The signature of `Callback.on_train_epoch_end` has changed in v1.3. `outputs` parameter has been removed. Support for the old signature will be removed in v1.5
"The signature of `Callback.on_train_epoch_end` has changed in v1.3."
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/tpu_spawn.py:192: UserWarning: cleaning up tpu spawn environment...
rank_zero_warn("cleaning up tpu spawn environment...")
```
### Expected behavior
The progrès bar must show up.
### Environment
- PyTorch Lightning Version: 1.4.9
- PyTorch Version: 1.8.0a0+6e9f2c8
- Python version: 3.7.10
- OS (e.g., Linux): Linux
- CUDA/cuDNN version: N/A
- GPU models and configuration: N/A
- How you installed PyTorch (`conda`, `pip`, source): pip
- If compiling from source, the output of `torch.__config__.show()`: N/A
- Any other relevant information: TPU on Kaggle
### Additional context
N/A
cc @kaushikb11 @rohitgr7 @tchaton
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the training setup on a Kaggle TPU with a DataLoader using num_workers greater than zero, then compare it with num_workers set to zero. Investigate the TPU worker and progress-bar output paths; done means the progress bar appears during training without breaking the successful run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100