bigscience-workshop / bigscience-workshop/petals

Random error when starting docker container again

Open
#468 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10.6k
Forks
650
PR merge metrics
No merged PRs in 30d

Description

When starting petals in docker, sometimes it will fail with the below error.

Steps:
1. Start petals with the below command
2. Exit petals with Ctrl-C
3. Restart the computer
4. Start petals with the below command again

Immediately re-running the command will succeed without error.

```bash
docker run --rm -p 31330:31330 --ipc host --gpus all --volume petals-cache:/cache --rm learningathome/petals:main python -m petals.cli.run_server --port 31330 stabilityai/StableBeluga2
```

Error message:

```
Aug 18 10:18:55.866 [INFO] Running Petals 2.0.0.post3
Aug 18 10:18:56.069 [INFO] A newer version 2.0.1.post2 is available. Please upgrade with: pip install --upgrade petals
Downloading (…)lve/main/config.json: 100%|██████████| 630/630 [00:00<00:00, 6.23MB/s]
Aug 18 10:18:56.764 [INFO] Make sure you follow the LLaMA's terms of use: https://bit.ly/llama2-license for LLaMA 2, https://bit.ly/llama-license for LLaMA 1
Aug 18 10:18:56.765 [INFO] Using DHT prefix: StableBeluga2-hf
Aug 18 10:19:11.835 [INFO] This server is accessible directly
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/petals/src/petals/cli/run_server.py", line 224, in
main()
File "/home/petals/src/petals/cli/run_server.py", line 208, in main
server = Server(
File "/home/petals/src/petals/server/server.py", line 135, in __init__
self.dht = DHT(
File "/opt/conda/lib/python3.10/site-packages/hivemind/dht/dht.py", line 88, in __init__
self.run_in_background(await_ready=await_ready)
File "/opt/conda/lib/python3.10/site-packages/hivemind/dht/dht.py", line 148, in run_in_background
self.wait_until_ready(timeout)
File "/opt/conda/lib/python3.10/site-packages/hivemind/dht/dht.py", line 151, in wait_until_ready
self._ready.result(timeout=timeout)
File "/opt/conda/lib/python3.10/site-packages/hivemind/utils/mpfuture.py", line 262, in result
return super().result(timeout)
File "/opt/conda/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/opt/conda/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
FileNotFoundError: [Errno 2] No such file or directory
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.