BUG: Without internet, neptune gets stuck in multi-gpu pytorch lightning and crashes the run

Nobody has claimed this yet.

### Bug description

### Describe the bug

If the neptune client loses connection to the neptune server for any reason, it waits forever on GPU0 to get the connection back. This crashes the training run at the end of the training epoch.

### Reproduction

Train anything with PyTorch Lightning in a multi-gpu settings.

Use neptune as a logger.

Have an internet connection that stalls forever (e.g. blocked ports).

At the end of the training epoch wait another 30 min until you get a NCCL TIMEOUT where the GPUs 1-7 get tired of waiting for GPU 0 and crash.

### Expected behavior

The client should somehow gracefully store data locally instead of tearing everything down.

### Traceback

If applicable, add traceback or log output/screenshots to help explain your problem.

### Environment

**The output of `pip list`:**

```
Package Version Editable project location
------------------------- -------------- -------------------------------
accelerate 1.4.0
aiohappyeyeballs 2.5.0
aiohttp 3.11.13
aiosignal 1.3.2
annotated-types 0.7.0
antlr4-python3-runtime 4.9.3
anyio 4.8.0
argon2-cffi 23.1.0
argon2-cffi-bindings 21.2.0
arrow 1.3.0
astroid 3.3.9
asttokens 3.0.0
async-lru 2.0.4
attrs 25.1.0
babel 2.17.0
beautifulsoup4 4.13.3
bitsandbytes 0.45.3
black 25.1.0
bleach 6.2.0
blinker 1.9.0
blis 1.2.0
boto3 1.37.9
botocore 1.37.9
braceexpand 0.1.7
bravado 11.1.0
bravado-core 6.1.1
cachetools 5.5.2
cairocffi 1.7.1
CairoSVG 2.7.1
catalogue 2.0.10
certifi 2025.1.31
cffi 1.17.1
cfgv 3.4.0
chardet 5.2.0
charset-normalizer 3.4.1
click 8.1.8
cloudpathlib 0.21.0
comm 0.2.2
confection 0.1.5
contourpy 1.3.1
coverage 7.6.12
cssselect2 0.8.0
cycler 0.12.1
cymem 2.0.11
dash 2.18.2
dash-bootstrap-components 1.7.1
dash-core-components 2.0.0
dash-html-components 2.0.0
dash-table 5.0.0
datasets 3.3.2
debugpy 1.8.14
decorator 5.2.1
defusedxml 0.7.1
Deprecated 1.2.18
dill 0.3.8
distlib 0.3.9
executing 2.2.0
fastjsonschema 2.21.1
fasttext-numpy2 0.10.4
filelock 3.17.0
Flask 3.0.3
flask-sock 0.7.0
fonttools 4.56.0
fqdn 1.5.1
frozenlist 1.5.0
fsspec 2024.12.0
ftfy 6.3.1
future 1.0.0
gitdb 4.0.12
GitPython 3.1.44
google-api-core 2.24.1
google-api-python-client 2.163.0
google-auth 2.38.0
google-auth-httplib2 0.2.0
googleapis-common-protos 1.69.1
greenlet 3.1.1
h11 0.14.0
h5py 3.13.0
httpcore 1.0.7
httplib2 0.22.0
httpx 0.28.1
huggingface-hub 0.29.2
identify 2.6.9
idna 3.10
importlib_metadata 8.6.1
importlib_resources 6.5.2
iniconfig 2.0.0
ipdb 0.13.13
ipykernel 6.29.5
ipython 9.0.2
ipython_pygments_lexers 1.1.1
ipywidgets 8.1.5
isoduration 20.11.0
isort 6.0.1
itables 2.2.5
itsdangerous 2.2.0
jedi 0.19.2
Jinja2 3.1.6
jmespath 1.0.1
joblib 1.4.2
json5 0.10.0
jsonpointer 3.0.0
jsonref 1.1.0
jsonschema 4.23.0
jsonschema-specifications 2024.10.1
jupyter 1.1.1
jupyter_client 8.6.3
jupyter-console 6.6.3
jupyter_core 5.7.2
jupyter-events 0.12.0
jupyter-lsp 2.2.5
jupyter_server 2.15.0
jupyter_server_terminals 0.5.3
jupyterlab 4.3.5
jupyterlab_pygments 0.3.0
jupyterlab_server 2.27.3
jupyterlab_widgets 3.0.13
kiwisolver 1.4.8
langcodes 3.5.0
language_data 1.3.0
lightning 2.5.0.post0
lightning-utilities 0.14.0
lmdb 1.6.2
loguru 0.7.3
lxml 5.3.1
marisa-trie 1.2.1
markdown-it-py 3.0.0
MarkupSafe 3.0.2
matplotlib 3.10.1
matplotlib-inline 0.1.7
maturin 1.8.2
mccabe 0.7.0
mdurl 0.1.2
memory-tempfile 2.2.3
mistune 3.1.2
monotonic 1.6
mpmath 1.3.0
msgpack 1.1.0
multidict 6.1.0
multiprocess 0.70.16
murmurhash 1.0.12
mypy-extensions 1.0.0
narwhals 1.29.1
natsort 8.4.0
nbclient 0.10.2
nbconvert 7.16.6
nbformat 5.10.4
neptune 1.13.0
nest-asyncio 1.6.0
networkx 3.4.2
nibabel 5.3.2
nilearn 0.11.1
nltk 3.9.1
nodeenv 1.9.1
notebook 7.3.2
notebook_shim 0.2.4
numpy 2.2.3
nvidia-cublas-cu12 12.4.5.8
nvidia-cuda-cupti-cu12 12.4.127
nvidia-cuda-nvrtc-cu12 12.4.127
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu12 11.2.1.3
nvidia-curand-cu12 10.3.5.147
nvidia-cusolver-cu12 11.6.1.9
nvidia-cusparse-cu12 12.3.1.170
nvidia-cusparselt-cu12 0.6.2
nvidia-ml-py 12.570.86
nvidia-ml-py3 7.352.0
nvidia-nccl-cu12 2.21.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu12 12.4.127
oauthlib 3.2.2
omegaconf 2.3.0
overrides 7.7.0
packaging 24.2
pandas 2.2.3
pandocfilters 1.5.1
parso 0.8.4
pathspec 0.12.1
peft 0.14.0
pexpect 4.9.0
pillow 11.1.0
pillow-avif-plugin 1.5.0
pip 25.0.1
platformdirs 4.3.6
plotly 6.0.0
pluggy 1.5.0
pre_commit 4.1.0
preshed 3.0.9
prometheus_client 0.21.1
prompt_toolkit 3.0.50
propcache 0.3.0
proto-plus 1.26.0
protobuf 5.29.3
psutil 7.0.0
psycopg2-binary 2.9.10
ptyprocess 0.7.0
pure_eval 0.2.3
pyarrow 19.0.1
pyasn1 0.6.1
pyasn1_modules 0.4.1
pybind11 2.13.6
pycocoevalcap 1.2
pycocotools 2.0.8
pycparser 2.22
pydantic 2.10.6
pydantic_core 2.27.2
pydub 0.25.1
Pygments 2.19.1
pyinstrument 5.0.1
PyJWT 2.10.1
pylint 3.3.5
pyparsing 3.2.1
pytest 8.3.5
pytest-cov 6.0.0
python-dateutil 2.9.0.post0
python-dotenv 1.0.1
python-json-logger 3.3.0
pytorch-lightning 2.5.0.post0
PyTurboJPEG 1.7.7
pytz 2025.1
PyYAML 6.0.2
pyzmq 26.2.1
referencing 0.36.2
regex 2024.11.6
requests 2.32.3
requests-oauthlib 2.0.0
retrying 1.3.4
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rich 13.9.4
rpds-py 0.23.1
rsa 4.9
s3transfer 0.11.4
safetensors 0.5.3
scikit-learn 1.6.1
scipy 1.15.2
seaborn 0.13.2
Send2Trash 1.8.3
sentencepiece 0.2.0
setuptools 75.8.2
shellingham 1.5.4
simple-websocket 1.1.0
simplejson 3.20.1
six 1.17.0
smart-open 7.1.0
smmap 5.0.2
sniffio 1.3.1
soupsieve 2.6
spacy 3.8.4
spacy-legacy 3.0.12
spacy-loggers 1.0.5
SQLAlchemy 2.0.38
srsly 2.5.1
stack-data 0.6.3
swagger-spec-validator 3.0.4
sympy 1.13.1
tabulate 0.9.0
termcolor 2.5.0
terminado 0.18.1
thinc 8.3.4
threadpoolctl 3.5.0
timm 1.0.15
tinycss2 1.4.0
tokenize_rt 6.1.0
tokenizers 0.21.0
tomlkit 0.13.2
torch 2.6.0
torchaudio 2.6.0
torchmetrics 1.6.2
torchvision 0.21.0
tornado 6.4.2
tqdm 4.67.1
traitlets 5.14.3
transformers 4.49.0
treelib 1.7.1
triton 3.2.0
typer 0.15.2
types-python-dateutil 2.9.0.20241206
typing_extensions 4.12.2
tzdata 2025.1
uri-template 1.3.0
uritemplate 4.1.1
urllib3 2.3.0
virtualenv 20.29.3
wasabi 1.1.3
wcwidth 0.2.13
weasel 0.4.1
webcolors 24.11.1
webdataset 0.2.111
webencodings 0.5.1
websocket-client 1.8.0
Werkzeug 3.0.6
wheel 0.45.1
widgetsnbextension 4.0.13
wrapt 1.17.2
wsproto 1.2.0
xxhash 3.5.0
yarl 1.18.3
zipp 3.21.0
zstandard 0.23.0
```

**The operating system you're using:**

Ubuntu 22.04

**The output of `python --version`:**

Python 3.12.9

### Additional context

Training on a SLURM cluster.

### What version are you seeing the problem on?

v2.5

### How to reproduce the bug

```python

```

### Error messages and logs

_No response_

### Environment

_No response_

### More info

neptune-client team asked me to move this issue to pytorch-lightning

https://github.com/neptune-ai/neptune-client/issues/1918

No source file or test is named. Start by reproducing the multi-GPU PyTorch Lightning run with Neptune logging and a stalled internet connection, then trace the logger behavior around the end of an epoch. Done means the training run does not leave GPU 0 waiting indefinitely or trigger an NCCL timeout, and queued data is stored locally as requested.

Written by the indexing model from the issue text.